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ABSTRACT 

This report describes research directed at designing 
and evaluating computer assisted instructional (CAI) systems capable 
ox inferring structural models of a student's reasoning strategies 
and identifying his underlying misconceptions. Several prototype 
systems using representative domains of knowledge were constructed. 
Erom these an inforiraticn processing framework compri:5ing models of 
expert reasoners^ adaptive tutors^ and students^ have evolved. 
Section 1 describes two paradigmatic instructional systems involving 
a decisior. aaking and a gaming environment. Section 2 explores issues 
of building intelligent instructional systems over more complex 
domains of knowledge. Section 3 describes research related to the 
design of robust intelligent systems. (Author/WBC) 
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This report describes research directed at designing 
and evaluating instructional systems which are able to use 
their knowled^5e to mimic some of the capabilities of a p.ood 
tutor such as being able to construct/infer structural 
models of a student's reasoning strategies and to identify 
his underlying misconceptions. Our basic research 
methodology has been to construct several highly modif iab le , 
prototype systems, each of which emphasizes some aspect of 
our intelligent CAI paradigm. By carefully choosing 
restricted (but representative) domains of knowledge for 
these prototype systems, an information processing framework 
comprising models of expert reasoners, adaptive tutors, and 
students has evolved. 

This report is broken into three sections. The fir^t 
section describes two paradigmatic instructional sysuems 
built around a decis.!on making ^nd a gaming environment. 
Each of these systems illustrates some of the needs and 
techniques for automatically inducing and using a structural 
model of the student *s reasoning strategies. The second 
section moves away from these elegant but simple domains of 
knowledge and explores some of the issues of building 
intelligent instructional systems over more complex domains 
of knowledge, such as remedial mathematics. The last 
describes some research related to the general issue of how 
to design robust Intel i ir^.ent systems. 
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INTiniDilL'TKiN 



We are at the thre^holci of a (iramatic advancement in 
(^-■^nputer technology which should change the way computers 
\ro omploved in instruction. This technological advance 
will decrease uhe cost of computer hardware to the extent 
that oach student will have available computational 
roso .r^cos which are currently restricted to a few elite 
users. The R;reatest challenge facing educational 
'i.echno 1 op; i St s is to productively harnesr' the increased 
availability of computational power to provide an equally 
dramatic improvement in the qual i ty and cost-effectiveness 
of instructional systems. Traditional comput er- ass i s te d 
instruction (CAI) paradigms were developed under the 
assumpt ^ ^^: that computational power is a scarce resource and 
these , i r-ad i i^!;ms are, for the most part, incapable '^f 
exploiting the latest technological advances. To 
effectively use the increased availability of computational 
power requires a re-evalua tion of the role of the computer 
in instructional paradigms. 

This report describes research directed at 
understanding and designinr^ instructional systems which take 
fuller advantage of the computational power afforded by new 
technology. This new kind of instructional system will be 
able to do more than spew forth its knowledge as factual 
information. It will be able to use its knowledge to form 
structural models of a student^s reasoning strategies, to 
determine his strengths and weaknesses, and to identify his 
r;] i sconeep t ions . Once it forms si^eh a model of the learner, 
it will then use this knowledre to determine when and how to 
provide remediation, heuristic recommendations ("hints"), or 
further instruction. To expect nn instructional system to 
make such decisions solely on the ba sis o f its own know l edge 
represents a substantial shift away from the basic notion of 
fixed instructional (linear or branching) sequences toward a 
TDre autonomous, truly adaptive, individualized instruction. 

Our basic methodology for understanding the design and 
operation of such instructional systems is to construct 
^>everal highly modifiable, prototype systems, each of which 
emphasizes some aspect of the overall adaptive approach. By 
carefully choosing restricted (but representative) domains 
of knowledge for the prototype system, we have begun to 
uncover a viable information processing framework of 
"intelligent" instructional systems, comprising models of 
expert reasoners, adaptive tutors, and students. This 
freneral framework will guide us in coping with less 
restricted domains of knowledge and in increasing the 
tutorial capabilities over the limited domains. 



!'fi..f \r. th" rr.-;enrcli described in this report,, we have 
.■etua.fuet ed CAl ■•,y:;t,(^ms that use their built-in in te 1 1 i Ronce 
to evnlu.-ite ?. student's answers and construct 
>'ounter-ex-)mples to his hypotheses <Brown et al 74>. From 
ther.(^ syste-^s, arose the need for tutorial initiative on the 
part of the system to interrupt the student and comment on 
-•,o(no railing in the student's behavior. For any but the 
mo-.t, trivial domain, it is clear that such interruption had 
to be based on an accurate "model" of the particular student 
in order to avoid inappropriate or irrelevant comments. One 
of the aims cf the research performed under this contract 
was t-he development of such models. From this research we 
discovered the difficulty of constructinf^ a system that can 
aiitomaticalll formulate a model of what the student is doing 
which succinctly represents his reasoning and knowledge to 
the extent necessary for isolating his fundamental 
•ni -.oonceptions. To complicate the devel^^pment task further, 
t.\\r "intelligent" adaptive tutor must then be able to take 
iivnntace of the synthesized information (e.g. structural 
"lodels of the student's reasoning and knowledge) in order to 
.-enerate po dacogi ca 1 1 y sound criticisms or hints. 

This rep-rt is broken into three sections. The first 
•.oetion (chapters one and two) begins with an explanation of 
the need for a fundamental change in point of view with 
re^^-rd to the design of complex CAI systems. The remainder 
>rchapter one and chapter two describe two paradigmatic 
instructional systems which were constructed to investigate 
(nnd ostabMsh the foundation for) a new viewpoint for CAI. 
Ln the oroposed v difrm the instructional system itself 
orpt.^lns an "intelligent" tutoring module capable of 
autonatically inducing and usine a structural model of the 
student's reasoning strategies. Chapter one describes^ a 
'.v-ter. to monitor a student's decision making activities 
ov^^r the domain of "attribute blocks". Chapter two 
ipsoribes a system to enhance the educational effectiveness 
of a Plato drill-and-practice game. In addition to the 
^utorin^ modules, both systerr.s contain expert 
oroblem-solving modules which assist th. tutoring module in 
oon-.tructing the model of the student and assist the student 
when help is requested. The subject domains of these 
-/stems were necessarily restricted to being structurally 
"i.ieal" or simple in order to expedite the investigation. 

The second section (chapters three and four) moves away 
'>om these elegant but siTiple domains of knowledge and 
-vpi-^res the problem of building intelligent adaptive 
^ r -.tructional systems over more complex domains of 
knrwledee, in particular the domains of electronics and 
>'<~-,.Pdial mathematics. Chapter three (on electronics) 
r-puses primarily on a '-gical theory of troubleshooting and 
-Pfers a precise information processing model of how an 
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oxj)or*t I. roub 1 (•^r>hQoter rcnu^onii. This model will play a 
fundamental role in the desif/n of any completo "intelligent" 
i n.struc tional system for teaching electronics via the 
troubleshooting methodology. Chapter four addresses the 
problem of teachine; procedural knowledge (as opposed to the 
axiomatic knowledge underlying basic electronics). For 
these initial investigations we have chosen the large and 
mostly unspecified procedural knowledge that underlies the 
ability to solve high school level algebra problems. By 
making explicit all the tacit or implicit knowledge 
comprising the procedural skills of algebra simplification, 
we will achieve the first step in creating an instructional 
system that can isolate and remediate "bugs" or mistakes 
contained in a person's interna l representation of these 
procedures (skills) . 

The last section (chapters five and six) describes some 
research related to the general issue of how to design and 
use intelligent instructional systems. Chapter five focuses 
on the use of the "intelligence" or knowledge base embedded 
in the instructional system to achieve a new dimension in 
man-machine communications. Chapter six sets forth some 
general guidelines for the design of intelligent computer 
assisted instructional systems. As such, this last chapter 
establishes a theoretical cornerstone for a wide range of 
instructional systems which can fully exploit the 
computational powers of tomorrow's computers in order to 
achieve higher quality adaptive instruction. 

A wor*d of caution to the reader. Each of these 
chapters contains many technical details of prototype 
systems which carry out their designated tasks. As 
Weizenbaum demonstrated with his FLIZA program <Weizenbaum 
66>, the external appearance of programs can be very 
deceptive and the merits of programs attempting something as 
complex as tutoring cannot be understood by its behavior. 
Much of what is novel and ''.n terest ing in this report is 
immersed in details of how tasks are performed. We 
therefore encourage the reader to pay more than passing 
attention to technical aspects of each chapter. Only those 
portions of the systems which illustrate either useful 
techniques or points of view on system organization are 
described. It is all too easy to talk a^ >ut structural 
models of a student and it is all too hard to understarJ 
what is involved in actually constructing them ev.?n over 
simple knowledge domains. 
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CflAI'Tl.;,^ 

AhAl'I'!^'' lN;;THl)(:Tri.)N ANI^ '"^^M Pi , |.; y K'NOWLL-lDGl' -BASHD CAI» 



In r^'-'^Mtt. yp-irs, tho cost computer hardware (central 
procosJ!Ln»J ^'ni tr, , memories ant-j ^, ^ike) has t-->een declining 
very rapi^' .V. xhe obvioua i f^stat ion of this is the 

■iramanlc '^^'o^^ cost of the pockRt calculator. it is now 

possible buy fairly Vppg^iie microcomputer (the 

Hewlett f^^ckard HP 65) in departf"^"^ stores for a price 
comparable to expensive ster^:,^ ^^^giver. There is every 
indioatlofi th^t this trench wi^^ continue, and that we will 
be seeing ^°'^Puter.s of much cro^ter power available in this 
price r^nKe. g^^p lars^p r.cale compi^^^'''^ ■ as the FDP-10 
will be available at a pri^c ^^itnin the budget of the 

avora.f^e i'Chooi system. As ^ ^gs^l^ , we can expect CAT 
systems to beoome more and moro compl^'^- In particular, it 
seems re^'SonabiP to expect ^ f^gater use of techniques 
developed -^n the field of ^^"^ ti f>j^ . gi intelligence, whose 
appl icabil ^ '-y ^as been limited by ^heir use of features 
available, until recently, in only the largest and most 
sophisticated computer installations- 

On the Whole, ^ ^AI sy^-'tem justifiable only if it is 
doing some^^hiri^ ^^^^ cannot be don e' e^'^^"'" ^ V well without the 
use Of a ^°ml^u^er. Thr- most ^^^qoe capability of the 
computer ability to ^^^^ very rapid decisions 

regarding "hat ^o do next. Becg^j^^ this, it is possible, 
in principle, design a system ^^^y^ an adaptive capability 
that responds ^gpi^iy to the nce^^ an individual student. 
In the ear'^y days of CAI, there ^ great optimism regarding 
this kind °i capability. In fact^ it was claimed that a CAI 
system woU^-d be able to provide c^'^^^alent of a personal 

tutor for ^^sry ytudent. In much cur""^"^ CAI, however, this 
capability hag somehow fallen ^ the wayside! This is 
nartly because humans are ^ood adaptive devices, in their 
own rashi°n- m f^ct, there ar.- systems such as TICCIT in 
which the ^'^aptiyg, component is i^ft entirely up to the 
user. HoWever^ it seems "^^aso^^^^e to suppose that there 
are situations where the computer is superior, and it seems 
reasonaM^ to gsk why such ^ises have not been developed 
extensiv iy- a ma.ior component answer lies in the 

pre.-.^btly accepted view of adaptive instruction. The 
pro: .em c ■■ adaptive i n s L ■'^uc t i ^gs classically been 
forn. i^at-- as the "branching problem" °^ specifying when, in 
an in&L-i' jctioriai seauence, a br'a^.^^^ g^ould occur. We will 
argue tn^t this is the wrong vie^ of the problem and that 
there is ^"'"ethine "ron;^ with the ^''hole notion of an 
" instructi°nal sgouence". 

(*)Th:.s chapter 13 a nreliminary ^^^ft of a paper by Groen , 
Brown and f^'Tton whi-h vi3^- ^'^(?sonte'^ by Groen at the AFRA 
conferenc"^' 



III,,,,,,,.,,,. ill I'uMJcr' 1.1 <(lr-<HMi At.k i ri.-.on (>t)>. T\\v t.;i;-.i- 
, VM- t ti-it V.h.^ i n;-,t.ru(-t.ion:i! ih-(um:>s.s .M)ulci bo hrokon <iown 
int.. , .-..MMi-n.-.^ of :^\.:u'of■. Kncti rA.u'c r o n u: t n fi of a r.rt o| 
I i ..,,1 -.V, to which thr .•it.urlont t- o r, p.Hi d- '.i . Tho :^yMtem ha't 

b.^■•^r <MMnpori.-nt,,". : a ti i .'-t.or'v , non.sir.UnK "1 

,...,„....•,,. „t -,1 in,, t ti,> rM':-.[ion:-,.'.'; irin<lr hy Hio rA-Ucivnl; n 
of th.> .-.tucirnt- • .-, If-nrniru' pr-o(M-;i,-, ; and a (iocMMiori 
nroo.. lu,-.- which .Minhlfd I, he My.-.t.MTi t, > Hooido, on_ the ba.-.is 
hi-t(H-y and Uio modrl, whi(!h rAimuli wore t,o bo 
.,;...■•,'.„. -1 dnr ine Uu- noxl -.lat^o. Inhrr-ont in the approach 
, MO n<M ion that, ^-ivon a precir.e onout^h model, and a 
.>„o,M'h criterion, an n,,tinM decinion procedure 
•>u ; ' i"' 1 o ijtid . 

A - AtHr<^on har. (ierc n ra le d in hi- more recent work, 
i ■' ■, rtsasonahle franiewnrk for- looking at vocabulary 
.::Vrr'",^' in 1 ,.'c-,r.ibly certain kindr- of drill and practice 
,'; .. ,, i, ,]oey not oxt.ond to mnr<> complex 31 tua t i on:; ! 
n,,.,.,, ,.■>' it leant three rear-onr for ttii.-; 

, , . ino ^impi ir.t.ic notion:-, of " :U i mu I in; " and "responr,o" lac 
li'io ■expre:'sive po'.;-- noodod to dcr,cribe large, complex 
vnuvledro :^,tr-ucture:: -'d tho ,,roee:-.:;es which manipulPto 

> i o-'el^-inr --i-riato n -i t tiorna I i - a 1 models for complex 
information processing t-nr-kn is extremely difficult and 
y no t t)e po:- 1 b 1 '-' . 

;> r*'- fl-w dia^rim r. o t -h 1 , wi t t, its strict ^low of 
-.,;ptpol i- not - ro;.li:-ti- representation o^ the 
•li'i'ntive pvstomr. employing more ec.nplex structures. 

'M ,-xa-nle of su-h a cornolex :-v.Ttorn is shown in Figure 
, . i '■■ i . 'taken 'ro^n a der-rription of a system f5r 

r,:^'.!'o -.rorramminf lanrna.'e PAflC. Clearly, it is 
; o 1- -.oparat'^ out the dyr^amior. implied by a diagram 
; \^ ot riPt .■^onuo-oe of stimulus and response 

on t V'hat wA nood are techniaues for making such a 

<,o that we o^n i. folate aspects of it, 

'"/r. r^,.-t ' '^e'- for I*, '^ot no desirable criteria and 

.:,;;V;o ".,a^ -■r-W-.-rv procedures for 'neeting these criteria. 
• 1 ' "th"' " ' if.e paired a:;sociate learning tasks ( for 

Vtre'si--ir" flowchart:^. -f Ki'^ure 1.1 were appropriate) , 
:::!:-.:;,Pchpr^- Ure able to .r'^r^t., within the framework of a 
:.i:,n'lo nara l--.-pati- r,cenar«io. The oroblem in creating 
■.../..^itW^ int-lli^-^ent -.-Al at • n- present time is to formulate 
'o:':h/; raT^trmaMo-:.-:-nar;o. The .ue:-,tion which must be 
.rod Po:-rivelv if thop. io to bo any rationale for the 
:,H.,.o;.^.p r^i .--v;=;to;-;- Lo whether such paradiematic 
■ ,. Tt'o- ;"-'rr;ninoer '^f this chapter will 
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t '»rt. i,ii:;t r'Mct. 1 i»na 1 1 



I'^'t. ♦'tMn I ru.' , on tJi(.' bash; ot* itie cur'Pent; hl.:it;ory 
wtiLcfi r.l.iinu Iii:; h; t.d bo pr'ei:ujnlea 



rpT'iUMit :;l Irnu hi:; t^o :;tiKlent 



Upd'Me hlL'.lorw by enlt.^i'jnfT the 
L : i :3 1 1 1 rn u 1 u an d e : ^ o n s e 



<^ Has 



stap;o N ol* ihe pr'ocor^r. been reached?^ 



Terminal^; I nst ru.: l lonal Session 



ow Dl:-ir!''v 



r 1 I nr. r. r u c 1 1 o na 1 F^y stem 




Figure 1.2 
BIP's Information Flow Diagram 



be devoted to exploring a specific instance that provides an 
example of what we mean by a paradigmatic system. 

A prerequisite for a paradigmatic system is a subject 

domain which has a simple and elegant structure. The domain 
must have a logical formulation which is both well-defined 

and easily specifiable. Its logical structure must also 

support natural mappings (analogies) into the kinds of 

complex and real world domains hat instructional systems 
are intended to handle. 

A domain which appears to be ideal for this purpose 
derives from that part of the world of manipulatory 
mathematics known as Attribute Blocks. Although Attribute 
Blocks can be used to explore a rich variei:y of interesting 
common sense reasoning principles, we will focus on just one 
application, a game which combines the notions of logic, 
decision making and hypothesis formation iv;to an interesting 
exercise on how to ask optimal questions and how to draw 
inferences from the answers. 



Description of the Game 
This 



deck of 
has three 
SIZE: 
COLOR 
SHAPE 



game is played with the 32 attribute 
attribute cards and 2 looped strings, 
attributes : 
small or large 

: red or yellow or green or blue 

triangle or square or circle or diamond 



blocks , a 
Each block 



There is one block in the set 
combination of the values of the 



of 32 for each possible 
three attributes . 



1 s 



The deck is made 
an attribute value 



1 , 
2. 
3. 
H. 
5. 
6. 



LARGE 
SMALL 
BLUE 
RED 

YELLOW 
GREEN 



up of 18 cards, 
or the negation 

7. TRIANGLE 

8. CIRCLE 

9. SQUARE 

10. DIAMOND 

1 1 . NOT BLUE 

12. NOT RED 



Written on 
of a val ue . 



13 
14 

15, 
16, 
17, 
18, 



NOT 
NOT 
NOT 
NOT 
NOT 
NOT 



each card 

YELLOW 
GREEN 
TRIANGLE 
CIRCLE 
SQUARE 
DIAMOND 



The student takes the two looped strings and overlaps 
them like: 



This arrangement of the looped strings has created four 
areas. Area 1 is inside the string on the left (loop A) and 
outside the string on the right (loop B) . Area 2 is inside 
loop B and outside loop A. Area 3 is inside both loops. 
Area 4 is outside both loops. 

The ..^res labelled Card A and Card B in Areas 1 and 2 
represent ' vo cards which the teacher chooses from the deck 
of cards. The student is NOT told which cards have been 
chosen. The object of the game is for the student to guess 
the attribute value written on these two cards. To do this 
the student chooses blocks one at a time and asks the 
teacher where the block 'Joes (according to the rule that a 
bl-ck is placed inside of loop A only if it satisfies tne 
value on Card A, and inside of loop B only if it satisfies 
the value on Card P, e.ff. if Card A=SOUARE and Card B=NOT 
BLLIF, then the Large Yellow Square goes in Area 3). The 
student continues choosing blocks, asking where they go, and 
placing them there until he believes he has placed enough 
blocks" to uniquely identify (i.e. deduce) what both of the 
cards are. 

What can a computer do for this environment? 

Manipulatory math tools represent, in our opinion, one 
of the best uses of simple, inexpensive technology that we 
have encountered. It was therefore with some trepidation 
that we considered contaminating this otherwise simple 
domain with "high technolc: y" . Of course, if it served our 
criteria for pa r ad i ^-ma 1 1 c systems, that might be enough 
ii'sti rication , but after watching numerous people use 
attribute blocks we felt there were many important tasks 
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that could be better accomDlishud (and in some cases, only 
accomplished) by having a knowledge bas- j CAI system. 



Protocols 



Instead of providing a theoretical description of the 
kind of facilities our system provides for this highly 
structured environment, we will provide three annotated 
protocols. Each protocol builds on the previous one and 
illustrates additional tutoring features which are realized 
by having the instructional system contain adui.tional 
information processing capabilities. However before ^.-e can 
meaningfully describe these protocols we must refer to the 
basic architecture of this system. Figure 1.3 illustrates 
the functional decomposition of the system into the modules 
referred to below. 



The first protocol stems from a relatively simple 
version of Figure 1.3 in which there are only three 
monitors, and a tutor which performs no mediation of the 
output of these monitors. The first monitor (who heavily 
utilizes the "expert") evaluates the student^s conjecture 
about what a card is and decides 1) if the conjecture is 
necessarily correct (i.e. do the blocks placements entail 
that card and only that card), 2) if it is consistent with 
the known information (i.e. block placements but that there 
are still other possibilities for the card), or 3) if the 
conjecture is inconsistent and if so selects a 
counter-example. The second monitor determines if the 
placement of the chosen block could have been deduced from 
the existing blocks, i.e. is it a logically redundant 
block. From a strict information theoretic point of view, 
such choices reflect i n fo rma tionl ess questions. Detecting 
such questions is an easy task for the monitor to perform 
but a difficult one for a human teacher. The third monitor 
determines when there is sufficient information (blocks 
already placed) to be able to deduce what one of the cards 
i s . 



These monitors are constantly watching the environment 
(i.e. student moves) and "wake up" the tutor whenever they 
detect something of interest, informing the alerted tutor of 
what it was that they found worth waking him up for. Note 
that even in this elementary version of the system the exact 
flow of control is not easy to specify since It depends on 
when and why a monitor interrupts the normal course of 
action of the system. 




F'-gure 1.3 
Diagram of Blocks System 



ERIC 
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PROTOCOL 1 
Attribute Block Game 



This is an example of a student usi. ^ a simple version of 
the Attribute Blocks system. Tne lines which begin with 
"* " were typed by the student. 



WELCOME TO THE ATTRIBUTE BLOCKS WORLD 

TWO CARDS HAVE BEEN CHOSEN. 

System selects two cards. 

CHOOSE A BLOCK 

* LARGE Y SQ 

The student tells the system which block he would like 
placed. The system will recognize any reasonable 
abbreviation of a block description. 

PUT THE LARGE 'YELLOW SQUARE IN A;:^'A 3 



System tells him where to place his block in accordance 
with what the cards are. ^ 



The monitor watching the possibilities space wakes up 
the tutor to tell the student a potentially surprising 
fact: There is only one possible theory left for Card 
B even though there isn't a single block in AREA 2 and 
only two blocks placed! This is an example of the type 
of monitoring human s are not very good at. 

CHOOSE A BLOCK 

* IS CARD B LARGE? 

Student forms a conjecture. In the non-computerized 
version of the game, this question had only two 
answers; yes and no. That is, th^„ conjecture is the 
same as the card or it is not. The tutor allows this 
to be answered in three ways: Yes, No, or Maybe. In 
this case LARGE is the only possible value Card B could 



CHOOSE A BLOCK 

* SMALL YELLOW SQUARE 

PUT THE SMALL YELLOW SQUARE IN AREA 1 
YOU SHOULD BE ABLE TO DEDUCE CARD B NOW. 




4 
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h-ivc; so the answer is yes. 



YES, THAT IS CORRECT. 

CHOOSE A BLOCK 

» IS CARD A SMALL? 

Student forms another conjecture. In this case, the 
tutor finds a counter-example to his conjecture so the 
answer is "no and here's why". 

NO. IF CARD A WERE SMALL, THEN THE LIRGE YELLOW SQUARE 
WHICH IS I.N AREA 3 WOULD BE OUTSIDE 0'.' LOOP A. 

CHOOSE A BLOCK 

» LARGE Y DI 

PUT THE I.ARGE YELLOW DIAMOND IN AREA 3 

CHOOSE A PLuCf; 

« IS CARD A Yr.'.i.Ol"^: 

The .5V. .-^n I forn;s another conjecture. In this case the 
tutor informs him that his "puess" is consistent with 
the kno-,vn information but that it is just a guess 
because there are still other possibilities. This is 
an example of "maybe." Note that a human teacher would 
have a hard time distingui ching this fact and would 
simply answer yes or no depending on whether Card A was 
Yel I'^w! 

IT COULD BF, BUT IT COULD ALSO BE NOT RED, NOT GREEN, 
NOT BLUE, NOT CIRCLE, OR NOT TRIANGLE. 




CH^^OSE A BLOCK 

« L R CI 

PUT THE LARGE RED CIRCLE IN AREA 2 

This oartioular block performs a oerfect split of the 
hyoothesis space, i.e. it rules out NOT GREEN, NOT 
BLUE, and NOT TRIANGLE. 



CHOOSE A BLOCK 

* L G CI 

PUT THE LARGE GREEN CIRCLE IN AREA ^ 
Rules out NOT RED. 
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CHOOSE A BLOCK 

» HELP 

CARD A CAN NOW BE 
YELLOW, NOT CIRCLE. 

CARD B CAN NOW BE 
LARGE. 

HELP calls the expert to generate the possibilities 
left for each card. Now the student must distinguish 
between the two. 



CHOOSE A BLOCK 

* S Y DI 

PUT THE SMALL YELLOW DIAMOND IN AREA 1 
YOU DIDN'T GAIN ANY INFORMATION BY THAT. 

Another monitor decides that the answer to that 
question or block placement was deducible from the 
known information. Since the block the student chose 
contained both possible values for CARD A (it is both 
YELLOW and NOT CIRCLE), it necessarily had to fall in 
CARD A's area (AREAS 1 or 3), and therefore couldn't 
give him any new information about the value of Card A. 



CHOOSE A BLOCK 

* L Y CI 

PUT THE LARGE YELLOW CIRCLE IN AREA 3 
YOU SHOULD BE ABLE TO DEDUCE CARD A NOW. 

This block contains the negation of one of the 
hypotheses for CARD A ( NEGATION (NOT CIRCLE ) ) = CIRCLE ; 
and holds che other one constant (YELLOW). This is the 
correct strategy for breaking down a hypothesis space. 
The tutor tells him that he did just that and now 
should be able to deduce the card. 



CHOOSE A BLOCK 

* IS CARD A YELLOW 

YES, THAT IS CORRECT. 

YOU HAVE NOW GUESSED BOTH CARDS. 

--which he does! 




Th i 
tutor, 
besides 
when the 
trie s t o 
whenever 
wa tching 
tutor , 
several 
i ) the 
particul 



r,t^("* on d 
In thi 

acting a 
tutor i 
get the 
the t 
f o r a s 

in cons 

st ratec: ' 
concept 

ar choic 



protocol reflects a slightly more versatile 
r, version, the tutor performs some services 
s a mouthpiece for a monitor. For example, 
s alerted that a card can now be deduced, it 

student to see why this is the case. Also, 
utor is alerted by the monitor, which is 
tudent to choose a redundant block, the 
ultation with the expert, invokes one of 
OS for trying to get the student to realize 
or meaning of redundancy, and ii) why his 
e wa s r edun dan t . 



PROTOCOL 2 
Fxamples of Tutoring 



These are examoles of the tutoring which would have 
transpired in protocol 1 usin^ the more sophisticated tutor. 
The lines which begin with and "> " were typed by the 

student, where represents a student's statement and 

represents a student's response to a tutorial query. 



Fxamrie when 



student doesn't deduce a card: 



CHOO.SF A BLOCK 

**LAPGE YELLOW SQUARE 

Put the LARGE YELLOW SQUARE 



I n 



AREA 3. 



Put 



**L 

Can 



Y SO 

the SMALL YELLOW SQUARE in AREA 1 ^ . r ^ 

The student should be able to figure out what Lard 



B is. 

another 



Put 
bl ock 



instead of 



SSI ng 




laces 



Y DI 

you tell 



me 



LARGE, SQUARE 



some possibilities 
AND YELLOW 



for 



rd B? 



The tutor asks the student 
possible that the student 
it because the blocks both 
the student gives three 



about Card B. It is 
h3S just not considered 
fall in loop A. When 
possibilities the tutor 
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evaluates each giving counter-examples to the ones 
which are incorrect. 

Card B could be LARGE, 

But if Card B were SQUARE then the SMALL YELLOW SQUARE 

which is in Area 1 would be INSIDE of Loop B, 

And if Card B were YELLOW then the SMALL YELLOW SQUARE 

which is in Area 1 would be INSIDE of Loop B, 

By taking the initiative, the tutor has gotten the 
student to observe that his reasoning i:i 
incorrect , 



Put the LARGE YELLOW DIAMOND in AREA 3 



Example of tutoring after placement of a worth''Gc3S block: 



The possible theories are: 
Card A can now be YELLOW or NOT CIRCLE, 
Card B can now be LARGE, 



**S Y DI 

The nionitor has noticed that the student is 
plac nf^ a worthless block so tne tutor is called. 
The tutor tries to get the student to realize that 
he could predict where that block will fall. 

Where do you think the SMALL YELLOW DIAMOND will go? 

> AHEA 1 

It's true. It could go there. 

Is there anyplace else the SMALL YELLOW DIAMOND could go? 

> AREA H 

If the student thinks there is someplace else it 
could go, he must have an incorrect idea of what 
Card A could be (since he already knows what Card 
Bis,) 

What value could Card A have which would cause 
the SMALL YELLOW DIAMOND to go in Area Hi 

> NOT CIRCLE 

If Card A were NOT CIRCLE, the SMALL i'ELLOW DIAMOND 
would go INSIDE of Loop A and NOT in Area 




What v-ilue "'''11, i carf^ A :,i\fe whi^^ wo^^^ cause 
thp CMaLL "^^-LLuw diamond to in Area ^"^ 

> NOT dIAHO^D 

If Card A ^^'^e MOT DIAMOND then ^h.^ ih^^K YEllO'^' DIAMOND 
which is •'•'■■ea 3 v^ouH be OUTSi,)£ Loop a 

What v-.lue "oui(^ card A have whl^^^ wc^J^? cause 
the SMALL "^-LLOw DIAMOND to ^^^g ^? 

> NONE 

Is there ariVPlace besides Area i the SMALL 

YELLOW DIAf^OND could >o? 

> NO . 

When there oniy one place ^ ^Ioq^^ .a" it s 

not helping' Vou gijess what the Cards ' 

Put th.;-. F-'ALi, Yellow diamond m ^^^^^ i 



Fxample of alternative tutorir^g strategy after a 
worthless ^Ioq^,. 



The possiD^^ theories are: 

Card A can "ow 5^ YELLOW or '^01 circlE- 

Card e can "^ow be LARGE. 



»»3 Y DiA 



An alternative strategy the tutor when called 
by the monitor is to make ^he student aware that 
l^is block is not splittincr his hypothesis space. 
The first step is to get him to verbalize the 
P°ssi5iiities. If he were'^"^""^ ^^l 
-counter-examples wouij given. there were 

^ore, the tutor would ^^^^ with the subset the 
^^"^ent gives. 

Can you te^^ rrie some possibilities for ^^rd A? 

> YiaiQW A'"Ji^ Not circle 

Both of thC'Se Qre possibilities. 

f^ext the tutor asks ^lis block splits his 

'hypothesis space. Since he believes it would, the 
tutor walks him through the deductions to show him 
^i^rerently- 

Will the 5MALL YELLOW DIAMOND eli^-^gte either of those 



lh(H)r'i«*r>? 

> YES 

Would the SMALL YELLOW DIAMOND go INSIDE or OUTSIDE 
of a" loop which had a card value YELLOW? 

> INSIDE 
Pi Rht 

Would the SMALL YELLOW DIAMOND go INSIDE or OUTSIDE 
of a loop which hrd a card value NOT CIRCLE? 

> I 
Ric^ht 

So you know that the SMALL ^ELLOW DIAMOND froes INSIDE of 
Loop A no matter which of those values Card A has. 

Put the SMALL YELLOW DIAMOND In AREA 1 
That was an in forma tion- less block! 



The third protocol reflects a substantially expanded 
set of monitors. In this version, there are monitors which 
watch f^r instances when higher- order information can be 
logically deduced (besides just the identity of a card). 
Such an example might be that, given the current placement 
of blocks, one can logically conclude that a card could not 
be any SHAPE. These more abstract cha^^ac terizations of the 
remaining possibilities for a card stem from knowing general 
rules or principles about the structure of this environment. 
For example, one rule about shapes is if there are two 
blocks of the same shape with one of them inside a loop and 
the other outside of a loop, then the card for that loop 
can't be either a positive or negative shape! This is a 
non-trivial abstraction and helping students to discover 
such abstractions is a worthwhile goal. This third version 
has monitors that understand all these abstractions and 
whenever one is encountered, it alerts the tutor who can 
then question the student much as was done in the previous 
protocol . 

In addition to these monitors, the expert's 
capabilities have been expanded to compute the ac t ual and 
ex pec ted information gain from any particular question (i.e. 
block choice). This information enables the student to 
experiment with the effectiveness of different decision 
strategies and enables the tutor to provide substantive 
hints as to what is the next best question to ask (i.e. 
block to choose). This facility in conjunction with the 
HELP command, which invokes the expert to compute all 
consistent possibilities for each card, provides an entirely 
new dimension to this environment ~~ a dimension that would 
be nearly impossible to realize without a computer. 
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PROTOCOL 3 
Information and Event Monitorino; 



Thi::r protocol is the same game that was played in 
Protocol 1, i.e. the same values for the cards and the same 
sequence of blocks. For this protocol however the 
information monitor and the event monitors have been turned 
on. The protocol also points out the use of the HINT 
commanci. The st ] nt typed the lines which begin with 
which is follcw'^d . / either a block choice or a request for 
"HELP" or a hint. 



Cards A and B have been c!:osen 

CHOOSE A BLOCK 

*»LARGE Y SO 

Put the LARGE YELLOW SOUAFE in AREA 3 

The expected information content of that block was 1.0 
The actual information content was 1.0 

The unit of information can be thought of "the 

number of blocks required to uniquely a ^line 

both cards". The best block generalx^/ has 
expected . formation of about 1. 



**SMALL YELLOW SQUARE 

Put the SMALL YELLOW SQUARE in AREA 1 

The expected information content of that block was .50 
The actual information content was 1.67 

Notice that in this case the student gambled with 
a "bad" block and won. The information monitor 
provides the student a new tool with which to 
compare strategies and choose between alternative 
ways of makinn: decisions. 

Below are the event monitors which were fired by 
the placement this block. In future systems, 

these monitors would invoke the tutor similar to 
the way the placement of a worthless block does 
now. Notice the number of event monitors which 
fiped — this confirms the fact that the actual 
information gained by this block was quite high. 



Card E must be size 
Cdrd B must be positive 
Card E can ^ t be shape 




Card B can't be color 
Card B can't be negative 
Card B can't be positive shape 
Card B can't be positive color 
Card A can ' t be size 
Card B can't be negative color 
Card B can't be negative shape 

**L Y DI 4 

Put the LARGE YELLOW DIAMOND in AREA 3 

The expected information content of that block was .41 

The actual information content was .21 

Card A can't be positive shape 

**HINT 

I think placing a NOT YELLO. CIRCLE OR TRIANGLE 
would be a good idea. 

The HINT comman d uses the in formation measure to 
determine the set of Blocks with the highest 
expected gain. An intensional description of this 
set is then calculated and given to the student. 
The intensional description allows the student to 
see which attributes are critical to the splitting 
of the hypothesis space. 




**L R CI 

Put the LARGE RED CIRCLE in AREA 2 

The expected information content of that block was 
The actual inforrration content was .5 

**HELP 

Card A can now be YELLOW, NOT RED or NOT CIRCLE 
Card B : an now be LARGE . 

**HINT 

I think placing a NOT YELLOW thing 
which is not CIRCLE, a BLUE OR GREEN 
thing or a NOT RED CIRCLE would be a good idea. 

**L G CIR 

Put the LARGE GREEN CIRCLE in AREA 2 

The expected information content of that block was .46 

The actual information content was .29 



Card A can't be negative color 
**HELP 

Card A can now be YELLOW or NOT CIRCLE. 
Card B can now be LARGE. 
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**S Y DIA 

Put the SMALL YELLOW DIAMOND in AREA 1 

The expectod information content of that block was 0.0 
The actual information content was 0.0 

I think placing a NOT YELLOW thing which is not CIRCLE 
or a YELLOW CIRCLE would be a good idea. 



^*L Y CIR 

Put the LARGE YELLOW CIRCLE in AREA 3 

The expected information oontent of that block was 

The actual information content was .5 



Card A must be positive color 

Card A must be color 

Card A must be positive 

Card A can' t be shape 

Card A can't be negative 

Card A can' t bo negative shape 



• 5 




**STOP 



It must now be clear that the logical information being 
computed for the tutor is approaching information overload 
and if the tutor were sinnply to pass it on to the student 
or, for example, query him each time a monitor deduced some 
fact, the student would collapse into a hysterical twitching 
heap! The tutor is in need of some guiding principles for 
determining which of this information is important to a 
par ticular student at a e;i ven moment. But how can the tutor 
make any rational decisions along these lines without having 
at his disposal a model of what might currently be important 
to the student? Here we have a definite need for a 
structural model of the student. In particular, the model 
should make explicit which rules or generalizations the 
student already knows, which ones he clearly does not know 
and which ones he has used sometimes incorrectly. 



using and 
monitor is 



There is a beautifully simple paradigm for 
constructing such a model. Associated with each 
a set of rules he could use to derive or achieve its 
particular p:oal (fact). Any time a monitor achieves its 
eoals, it need inform the tutor not only of his success 



;ard A can't bo any SHAPE) but must 
tutor of the way It deduced this fact 
necessitate additional -alls on the expert.) 



also inform the 
,* (This might 
Then, the tutor 



(*)For the moment let us assume this derivation is unique 
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can either decide that the student has already shown mastery 
of these rules, or if in doubt, he can decide to query the 
student about the conclusion. If the student answers the 
question correctly then the tutor has evidence that he can 
successfully use these rules to derive the appropriate 
information. Whether the student answers the question 
correctly or not, the tutor has gained new information about 
which rules he knows and how he can conbine them. (Of 
course, there may be higher-level considerations that 
dissuade the tutor from asking any question at the moment 
that a monitor discovers something.) 

Production Rules : 

The rules comprising the structure of this environment 
can be viewed as production rule schemes in the sense of 
Newell and Simon. Similar formulations are currently being 
exploited by many cognitive psychologists. As such, the 
blocks world pr'ovides a nearly open ended range of 
possibilities for examining how to induce production rule 
based, structural models of a student. 

System Description 

The Attribute Blocks system has been structured to 
allow experimentation with various tutorial and assistance 
modules. In this section we will describe the overall 
structure of the Blocks system, and the workings and 
responsibilities of the individual modules. Figure 
shows the basic organization of processes and data within 
the Blocks system. 

Execut ive 

The executive has responsibility for the control flow 
within the system. The typical control path to process a 
student's move is as follows: The executive reads the input 
statement from the student and passes it to the natural 
language uiiders tander . The natural language understander 
identifies the intent of the statement and returns its 
semantic structure. The semantic structure contains the 
pertinent information for one of the environment maintainers 
(Unless, of course, the statement was directed to the tutor 
or one of the monitors.) The executive calls the proper 
maintenance routine which carries out the proper change to 
the environment. Then, before the student is told the 
result of his statement (e.g. where a particular block 
goes), the "pre-answer" tutor is called. At this point the 
tutor knows the student's move and what the result of that 
move will be. The tutor can, for example, query the student 
about what he had in mind by placing that block (while the 
student still has it in mind) or he can point out some 
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thitiK'S thai Iho student should have known but didn't 
(becauise IV he had, his present statement would have been 
different). After the tutor has finished, the executive 
calls the natural languap;e generator to tell the student the 
result of his statement. Then the tutor is called again, 
this time to further explain the results of the statement or 
possibly the ramifications of it. This is also when the 
various monitors are invoked. 

The executive also maintains the history list of 
student interactions. For each interaction, the student's 
statement, the result generated by the system, any advice or 
tutoring he was given and the context" in which the statement 
occurred are saved on the history list. The^history list is 
used by the tutor to find old possibilities lists and also 
to check on the types of tutoring it has given the student 
in the past . 

Natural Lanp;uage Understander 

The classes of sentences required by the Attribute 
Blocks system to date have been fairly straightforward and 
are handled easily by a small semantic grammar based 
processor <Brown & Burton 75>. The Blocks grammar has about 
15 semantic categories and involves very little complexity. 
However the flexibility allowed by the goal directed nature 
of the semantic grammar was particularly useful in the. rules 
for recognizing descriptions of blocks. Without it, the 
understander would have been much harder to write. 

In writing the parser for Blocks, the semantic grammar 
framework was extended by the aadition of commands. 
Commands are one word directives to the system such as HELP, 
HINT, STOP, SHUT/UP, EXPERIMENTING, etc. A facility was 
added to the parsinR; mechanism to allow words to be defined 
as commands and then recognized in a bottom-up manner which 
short-circuits the regular top-down parsing scheme. This 
facility has allowed new commands to be added quickly and 
easily. 

Environment maintenance 

The Blocks system is built around an environment of a 
student playin^x with attribute blocks. This environment 
consists of the values cf the cards, the locations of blocks 
which have been placed, and the possible theories for the 
cards which are consistent with the placed blocks.* The 
tasks involved in the Blocks laboratory are performed by 
procedural specialists called environment maintainers. 

<'*)The list of oc si b i 1 i t i o s can be recalculated from the 
blocks but was deemed important enough to make it part of 
t he env ironment . 
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These tnsk.s include placing and removing blocks (including 
determining where blocks should go) printing the present 
board configuration, setting up the cards to begin a game 
and stopping the session. The effect of each oC these 
maintenance actions is to change some portion of the data 
base which is examined by the other modules. 



Moni to rs 



To study the effects of various types of services which 
the Attribute Blocks laboratory could provide, several 
different types of monitors were designed and implemented. 
Protocol 3 presents the same example that was presented in 
Protocol 1 but with different monitors in effect. 



Remaining possibilities monitor 



In order to allow the student to see the effect that 
placing certain blocks had on the theory sets for each of 
the cards, a monitor was written which calculates all of the 
possible values for each of the cards from a configuration 
of blocks. Using this monitor, together with the ability to 
remove blocks, the student can discover how certain blocks 
(Questions) will split a set of possible theories. This 
monitor car also wake up the tutoring routines when 
worthless blocks are placed or when a set of possible 
theories is reduced to one element. 



Information Gain monitor 



The Attribute blocks world is an excellent domain to 
study problems of making decisions such as what makes a good 
question. The expected information gain of a block and the 
actual information gained provide a valuable metric for 
evaluating alternative questions. The expected information 
gain of a block is the sum over the four areas of the amount 
of information gained by that block falling in that area, 
times the probability of it falling there. The amount of 
information gained from a block falling in an area is the 
logarithm of the percentage of possible theories that block 
eliminated (in the cross product of theories for card A and 
theories for card B) . The logarithm is taken base 4 since 
each question has four possible answers (each block could go 
in one of four possible areas). Since the beginning theory 
space has 324 members (I8xl8), the expected number of 
questions required to isolate one individual element is 
LOG 324 (base 4) or about 4.2. When the total actual 
information gain of the student's blocks totals 4.2, he can 
deduce both cards. By seeing the expected and actual 
informatio:: gain, the student can begin to develop 
intuitions Toout "good" questions. 
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p:vent monitor^-^ provide a moans of monitoring the 
rerriaininK list of possibilities for the occurrence of a 
particular event. An evei^t occurs when a generalized class 
of values must be the case or can't be the case. For 
protocol ^ there were classes monitored for size, shape, 
color, positive shape, positive color, negative shape, 
negative color, positives and negatives. A class is defined 
as' a list of theory values. For example the positive shape 
class is (TRIANGLE CIRCLE DIAMOND SQUARE). Each time the 
student places a block, the new possibilities list is 
checked for a change of status with respect to each of the 
classes. There are two changes of sta'tus which are 
considered worthy of note. One is when the possibilities 
list no longer intersects class. In this case, that class 
has just become impossible. The other important change of 
status occurs when the possibilities list becomes a subset 
of one of the classes. In this case, that class has just 
become required. In protocol 3 the monitors merely evoke 
printing functions which herald the event to the student. 
In a more ;ull\ developed system, these monitors would 
invoke the tutoring component as is currently done when a 
worthless block is placed. The tutor would tliexi have the 
option of exploring the reasons for the event with the 
student . 

Pre-answer tutor 

The '*pre-answer tutor*^ springs into action when either 
(1) the student has olaced a worthless block or (2) the 
student has placed a block when he should have been able to 
deduce a card but didnU. The tutor attempts through a 
series of pre-stored questions to direct the student's 
attention to aspects of the situation that he may have 
missed. Protocol 2 shows several examples of this tutor 
intervening in a student's session. At present the tutor 
has three strategies, (1) If the student fails to deduce a 
card, try to get him to say what he thinks it could be and 
show'him by counter example where he is wrong. (2) If the 
student places a worthless block, try to get him to predict 
where it will go and convince him that is the only place it 
could go. (3) If the student places a worthless block, get 
him to verbalize some remaining theories and choose a block 
which would distinguish between them. 

From experiments, we have found this tutoring to be 
v'^uable although at present much too oppressive! When to 
tutor and when not to is a very difficult problem whose 
solution will require a structural model of the student. 
Th^-^ following chapter is devoted to precisely this issue. 



CHAPTER 2 

AN INTELLIGENT TUTORING AND STUDENT MODELLING SYSTEM 



In this chapter v/ ^ describe a paradigmatic CAI system 
that was built to investigate the problems of 1) developing 
a representation for a logically adequate model of a 
learner, 2; constructing actual models of learners in 
learning situations, and 3) constructing a tutor which could 
use the constructed models to provide (on its own) 
well-timed, insightful comments. The interactions between 
these three problems dictate that they be attacked 
simultaneously. The logical adequacy of a student model 
cannot be investigated without specifying how the model is 
to be used by the tutoring system. Likewise, there is no 
point in inventing a representation for a model which is 
structurally more complicated than one which the system can 
automatically induce . 

In classical CAI, the tutoring behavior is locally 
controlled by a predetermined instructional branching 
sequence which, at best, references a coarse model of the 
student. This differs substantially from our view, in which 
the tutoring module has complete freedom to interrupt the 
student at any time and must use its knowledge of the domain 
together with the synthesized model to decide what to say 
and when to say it. The viability of this approach depends 
critically on how well the model represents the student's 
reasoning strategies and current state of knowledge. If the 
tutor is to deviate from a predetermined instruc'tional 
sequence, its new course of action must be based on its 
reasoning capabilities and the minute details of a student's 
strengths and weaknesses! 

In order to gain some leverage on building a system 
that could actually construct and u?e a model of the 
student, we chose a domain in which we could easily 
construct an expert program that the tutor could call on for 
evaluating the student's behavior. The domain of knowledge 
chosen was the PLATO game "How the West Was Won." A 
provocative doctoral thesis by Cecily Resnick <Resnick 75> 
describes some preliminary experiments which question the 
effectiveness of this game as a learning environment. 
Taking her thesis as our starting point, we attempted to 
transform this arithmetic game into a highly productive 
learning environment by adding a student modeller and an 
intelligent, sensitive tutoring program . The t uto r ' s 
comments were to be sufficiently insightful and well-::imed 
that students continue to view the game as exciting and fun 
but at the same time actually learn something! The problem 
in building this system was to make the tutor neither too 
vocal — so that it would neither be constantly babbling i?t 
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the mout.h, de:5troyinK the appeal of the game, nor so 
reticent that little would actually be learned. We felt 
that this could be achieved by bringing powerful information • 
processing techniques to bear, thereby opening up an 
exci^,ing new domain of CAI. 

Description of "How the West Was Won"* 

This game is played with two opponents (the computer 
usually being one of the opponents), on a gamc^ board like 
that in Figure 2.1. The object of the game is to get to the 
last town' on the map (position 70). On each turn a player 
pets ^ spinners (random numbers). He can combine the values 
of the spinners using any 2 (different) arithitietic operators 
{+ _ * or /). The value of the arithmetic expression he 
makes' is the number of spaces he gets *-o move. (He must 
also specify the answer.) If he makes a negative number, he 
moves backwards. 

Along the way there are shortcuts and towns. If a 
Dlaver lands on a shortcut, he advances to the other end 
(e i from 5 to 13 in Figure 2.1). If he lands or a town, 
he Voes on to the next town. When a player lands on the 
same'place as his opponent, unless he is in a town his 
opponent soes back two towns. To win, a player must land 
exactly on'the last town. Both players get the 5 -..me number 
of turns, so ties are possible. 

Why Tutor at All? 

A central premise of complex, knowledge-bas- d CAI is 
that POod tutoring can point out structure in an environment 
which'might have otherwise been missed, and bv_ so doing 
allows the student to enrich his undcrstanaing of (and 
skills in) the environment. In West, an untutored 
(unwatched) student may tend to become fixed on a subset of 
the available moves and hence miss the pote-.-^al richness of 
the same. For example, a student may adopt the strategy of 
adding the first two spinners and multiplying the result by 
the third spinner, (A-fB)*C. Since the third spinner tends 
to be the largest, this strategy is close to the strategy of 
•nultiplying the largest number by the sum of the other two 
numbers (which produces the largest possible number). If 
this strategy is augmented by a rule that prevents moving 
off the board (i.e. a simple end game strategy) it 
.generates a respectable game. Notice, however, that much is 
missed. The student is unaware of the special moves such as 
bumps and therefore of such questions as, "Is it better to 
send my opponent back 1^ or get 9 ahead of him?" In fact, 

'»)This'game was written by Bonnie Anderson for the PLATO 
Flement-ry Mathematics Project. 

o 4 
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FIGURE 2.1 

Game Board for WEST (from PLATO terminal) 



LOCOMOTIVE'S Turn: 

Your numbers: 12 6 
Your move*^ 6x{l+2) = ♦ 1 8 

-NEXT- 



1 ID 1 

a 'Q .'o 



3 2 5 4 



Tomb- 0 
Stone 



1 2 3 4 5 6 7 8 9 \ 



Z' 19 18 17 16 15 14 13 12 11 



10 



-T)eath 
Valley 



Dodge 



20 



21 2 2 23 24 2 5 26 2 7 28 



^ 39 38~ 37 36 35 34 33 32 31 



29 \| 



30 Dry 
Gulch 




59 58 57 56 55 54 53 52 51 



50 Santa 
Fe 



Urbana .60 



61 62 63 64 65 66 67 68 69 




70 Red- 
Gulch 



sirUM-^ t.hc :;t,u(lonl Konorat(^r> only one move, ho misses the 
whoLe notion of stratri-ues for deciding betwet?n alternntivo 
moves. From an nrithmetic point of view, he is performing 
one calculation per move Instead of the dozens which he 
would have to perform to answer questions such as, "What 
numbers can I form with those spinners?" or "Can I make an 8 
with these spinners?^* By in ter ject in? comments and 
suR;gestinr^ better moves to the student from time to time 
(not too often), the tutor tries to widen the student^s view 
of the game . 

Tutoring by Issue and Example a General Paradigm 

The paradigm of "issues and examples" was developed to 
provide the tutoring system with the means to focus on 
relevant oortions of the student^s behavior. The important 
aspects (skills) of the domain (i.e. what the student is 
expected to know or learn) are identified as a collection of 
"issues". The issues determine what parts of the student's 
behavior are being monitored by the tutor. The issues are 
implemented as procedural specialists which watch the 
student's behavior for evidence that the student uses or 
does not use their particular concept or skill. As the 
student plays, a model of how he is performing, with respect 
to each issue, is constructed. V/hen he makes a "bad" move a 
tutorial component uses the model to decide why the student 
did not make a better move, that is, which issue he missed. 
Once an issue has been determined, the tutor might decide to 
present an explanation of that issue together with a better 
move wliich' illustrates the issue. In this way, the student 
can see the usefulness of the "issue" at a time when he will 
be most receptive to the idea presented — immediately after 
he has thought about the problem. 

Fie:ure 2.2 presents a diagram of the mod el 1 ing/ 1 u to r ial 
process underlying the paradigm. The first major component 
of the process is the construction of a model of the 
student's behavior. The model is constructed from an 
environment in which the student is solving a problem (in 
this case, a move in a game). Within this environment the 
student exhibits a certain behavior (such as making a move). 
The important aspects of this behavior (the issues) are 
abstracted into the model by the issue "recognizers". This 
abstracting is also done with respect to the behavior of a 
computer-based "Expert" in the same environment. The two 
abstractions are compared to provide a d i f ferential model of 
the student's behavior, which indicates those issues on 
which the student is weak. Notice that without the Expert 
it is not possible to determine whether the student is weak 
in some area or whether the need for that skill has arisen 
Infreouently in the student's experience. 
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Figure 2.2 
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Thf' MtMMwxJ major com[)orv-^nl of FIkihv^ .\ ? i tJi^^ 
"Tutor'". Wh<ni Iho^ nt.inicnt. makes a let\r. than 0()li7ial. movr 
(ao dotorminod by ^:omparinp: his move wit.h thoso of tht* 
[•ixport) the Tutor uses the is'^ue "eval ua to rt^" which scm the 
stU(UMit model to create a i'lt^t of issues on which the 
student is weak. From the B.xpert's list of better moves, 
the tutor* uses the "issue" recognizers to determine which 
issues are illustrated by better moves. From these two 
lists (the "weak" is'^uos and the better move issues), the 
tutor selects an issue and a ^ood move which illustrates it. 
The selected issue and example are then passed to the output 
k^enerators which produce the feedback to the student. 

We would Like to stress two points in the above 
process. One is the necessity of the Expert, and the other 
is the importance of identifying the critical issues. The 
!-:xpert provides a measure for evaluating the student's 
behavior in un pred icted situations. The issues define those 
struf^tured or conceptual components of the environment which 
the st:..,dent Is expected to learn and they provide a handle 
to structure and direct the exploration of the environment. 

Pro tocol 

Before discussing the mod ell ing/ tutor ing process in 
iireat^r detail, wo present in Figure 2.3 a protocol of a 
student playin.p; WEST. The tutoring component has been 
accelerated to (:!;enerate more feedback than normal. In 
particular, the normal paced tutor seldom hassles the 
student two moves in a row. In the protocol, all lines 
typed by the student begin with "=> '^ The lines which are 
indented 5 spaces are information determined by the model or 
the tutor and are not seen by the student. Annotations are 
i n den ted . 



FIGURE 2. 3 



Welcome to "How the West Was Won". 
What is your name? 
=> Debbie 

Do you want to go first? 
= > Yes 

It's YOUR turn.. Spinners 
What arithmetic expression 
=> 1*2+2 

What: number does that ^ive 



are : 12 2 
did YOU form? 

you? 

3S 
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TtH* nninbtw:; t*oi.ld haVf^ Immmi [d.-kIc wtMU^ : 

-1 -W, 0 () 1 
"bent" movo is (;-*;?)-f1. 

delta-ways: 10-8, 6-1, i)-^), V?, , 

Move r'anks 11 wtiioti i. FAIR 

The indent (Ml in format ion above is generated by the 
KXPKRT module on each turn. This information is not 
seen by the student but Is used by the TUTOR, M0DP:L 
KVALUATOR and STUDKNT MODELLER modules. It is 
included here to indicate the complexity of 
information that is used by the system to follow the 
student's btnavior. The first line gives the raiige 
of numbers that ouid legally be made usin;; the 
spinners. The secona line gives the "best" move 
according to the ^lotric of maximizing the difference 
between you and your opponent. (These two pieces of 
information are available to the student through the 
"hint" key if she wan ts it.) The third 1 ine 
(beginning "delta-ways") represents the legal move 
space. The first number in each pair gives the 
number of spaces aheat the student would be (i.e. 
delta). The second number is the number of different 
arithmetic patterns which could have been used to 
achieve that lead. In this case there is one move 
which would have put Debbie 13 ahead (making 5 and 
taking the shortcut), eight moves which would have 
put her 10 ahead (making 0 and getting advanced to 
the next town), one that would put her 6 ahead, etc. 
Her move (^4 ahead) ranked 11 because there were 10 
better ones, 1+(2*2) Is considered the same as 
(2*2)+l when determining the rankings. For the rest 
of the protocol only the Rn^ik of the student's move 
wi 11 be shown . 

Debbie is at ^ the COMPUTER is at 0 

It's MY turn.. Spinners are: 1 2 3 
My expression is: (1*2)+3 
Which gives ME 5 
I took a SHORTCUT 

the COMPUTER is at 13 Debbie is at l\ 

It's YOUR turn.. Spinners are: 2 1 ? 
What arithmetic expression did YOU form? 
=> 2*1+2 

What number does that give you? 
= > H 

Debbie makes another less-than-optimal move but two 
moves is ^oo early to start tutoring. 

Move ranks 9 which is FAIR 
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r»rhl> 1 - i I M t. Ur t:(^Ml'il I'KK 1 .m V 1 ^ 



1 1. ♦ MY Iwrn . . 'U^i^ nnorr^ n : 1 1 
My X pr o r» r. i on i : ( 1 -f 1 ) * < ' 

r.he COMIUJTKK i n t, ly Ue-'bbio i 



1 1. * r. YO t ur n . . 'ip i nn (m^ n o : 1 1 
WJiat ru' i thmrt. i <^xprM^"»3ion did YOW Tor^m? 

What nurnbor door. that, p/ivo you? 

Movp r'anl<r> < Which i HOOD 

Tho ISSIJK.S aro (PATTERN PAHKKJ STHATKi..') 



The so aro the 
do term i nod by 
with the movoG 
t hpoe movo.s h 
v;hich (\oor> not 
move u:'a.vs a 
pa rent, horror. 
not iced . The 
is more -'peeifi 
i nd i ca t*.'^' that 
for the student 



tiitorablo issues which have been 
consider in g the student model together 
which are better. Debbie's first 
ave all had the same pattern, a*b+c 
require parenther>os; whereas the best 
different pattern which does require 
tho issues of PATTERN and PARENS are 
issue of PATTERN was chosen because it 
c than PARENS. The STRATEGY issue 
there is no cohere nt pattern arising 
's moves, as will be discussed later. 



It's beginninp; to seem iike you stick to the same old 
standard pattern in your move. The really expert players 
try a whole bunch of sra b i 1 i t i e s . An example of a good 
expression would be: (1+1 )*5, which is a pretty good move. 
So vou could have been at 18 with ME finishing up the turn 
at 17. 

After the presenta tiotj of a better move, the student 
may be allowed to retake her turn. This usually 
makes the student more receptive to the advice. In 
this case Debbie decides not to put herself in front 
of her opponent . 

Would you like to take your turn over? 
= > No 

Debbie is at 1^ the COMPUTER is at ^^ 

It's MY turn.. Spinners are: 3 3? 
My expression is: (3-f2)*3 
Which gives MF 15 

the COMPUTER is at 32 Debbie is at 1^ 



It's YOUR turn. 
Whai arithmetic 



Spinners are 
expression did 



3 

YOU f 



rm , 



0 



What number does that give you? 
= > 23 

Move ranks 7 which is FAIR 

The ISSUES are (PARENS ORDER STRATEGY) 

In this case the tutor picks on the issue of 
parentheses use. ORDER (using the spinner in a 
particular order) was not a issue last turn because 
none of the better moves required changing the order. 
(In a less condensed session, the tutor would not 
comment on two moves in a row.) 

I notice that you don't use parentheses very often. I think 
you could win more often if you tried using parentheses. 
For example, you could have made the expression: (3+4)*5, 
which is alright for an amateur! So you could have been at 
49 and I would only have been at 32. 

Would you like to take your turn C7er? 
= > Yes 

In this case Debbie retakes her turn. The players 
are put in their original position and the spinners 
are the same. Retaking a turn allows the student to 
go thru the motions of the suggested move for 
themselves and makes it easier to remember. 

Debbie is at 14 the COMPUTER is at 32 

It's YOUR turn.. .Spinners are: 3^5 ^ 
What arithmetic exr^^ession did YOU form? 
=> (3+4)»5 

What number does that give you? 
= > 35 

Debbie is at 49 the COMPUTER is at 32 

It's MY turn.. Spinners are: 2 2 1 
My expression is: 2*(1-2) 
Which gives ME -2 
I landed in a town . 

the COMPUTER is at ^0 — Debbie is at ^9 

It's YOUR turn.. Spinners are: 1 2 3 
What arithmetic expression did YOU form? 
=> 2+1*3 

What number does that give you? 
= > 5 

Move ranks 13 which is POOR 

The ISSUES are (DIVIDE MINUS STRATEGY) 

Up to this point the only operations Debbie has used 
are addition and multiplication, so division and 
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subtraction become issues because there are better 
moves which use them. 



You shouldn't forget that it's alright to divide, so long as 
the answer comes out evenly. Especially when there is a 
good move not too far away, you should consider division as 
a way to make the right number. This expression is a 
typical example: (l+2)/3, which would have resulted in a 
TOWN! Then you would have been at 60 while I would have 
ended up at 40. 

Debbie is at 54 the COMPUTER is at 40 



It' s MY turn. 
My expression is : 
Which gives ME 4 
I took a SHORTCUT 
I bumped YOU 
the COMPUTER is at 54 - 



Spinners are: 2 4 2 
2+ (4/2) 



Debbie is at 40 



It's YOUR turn,. Spinners are: 1 2 2 
What arithmetic expression did YOU form? 
=> 1+2*2 

What number does 
^> 5 

Move ranks 1 1 which is POOR 

The ISSUES are (BUMP SHORTCUT ORDER STRATEGY) 



Spinners are : 
expression di d 

that give you' 



You don't seem to be bumping very much. Bumps are hard to 
get but are usually a good idea. One good example would be 
the expression: (l*2)+2, which would give you a SHORTCUT 
and a BUMP!! So you could have been at 54 while I would 
have ended up at 40. 

Debbie is at 45 the COMPUTER is at 54 

It's MY turn,. Spinners are: 1 2 1 
My expression is: (1+1 )*2 
Which gives ME 4 ^ 

the COMPUTER is at "58 Debbie is at 45 



turn.. Spinners are: 2 0 7 
What arithmetic expression did YOU form? 



The hint 
provid es 
First it 
thereby 



command (it is actually a single key) 
the student with three levels of help, 
tells the student what numbers are possible 
allowing him to pick one for himself. If he 



wants more help, the system will pick one for him. 
Finally the system will tell him what expression 
would make that number. Some students use the first 
level of hint as a check that they have though of all 
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the numbers. 

=> Hint 

The numbers you should be able to make are -1^1 -Q -2 2 
5 9 -7 7 U 0 -5 

What arithmetic expression did YOU form? 
=> Hint 

I think moving 5 would be a good idea. 

What arithmetic expression did YOU form? 
=> Hint 

You can make 5 by the expression (0+7)-2 

What arithmetic expression did YOU form? 
=> 2*0+7 

What number does that give you? 
= > U 

The WEST system also contains a simple arithmetic 
expression diagnostician which looks for mixed up 
precedence , 

MULTIPLICATION is done before ADDITION so 2*0+7 is equal 
to (2»0)+7 not 2*(0+7)- 

Would you like to change your expression? 
= > Yes 

Spinners are: 2 0 7 

What arithmetic expression did YOU form? 
=> 0+2*7 

What number does that give you? 
= > U 

Move ranks 3 which is GOOD 
Debbie is at 59 the COMPUTER is at 58 

It's MY turn,. Spinners are: 3 1 3 
My expression is: (1+3)*3 
Which gives ME 12 

the COMPUTER is at 70 Debbie is at 59 
I win , 

The COMPUTER has . won 382, lost 26 and tied 98 
Debbie has won 0, lost 1 and tied 0 

Hope we can play again sometime. 
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As stated earlier, the issues define those aspects of 
the environment which are abstracted into the model. The 
issues currently modelled over the West domain are: 

order of spinners the spinners don't have to be used in 

any particular order. 

parentheses: the use of parentheses is allowed 

and frequently valuable. 

backwards: if the result of a move i^ negative 

the player moves backwards which 
can sometimes lead to a special move. 

special moves: trying for towns, bumps, shortcuts 

is part of a good strategy. 

subtraction: subtraction is legal and sometimes 

use f ul . 

division: division is, legal and sometimes 

use f ul . 

pattern: the operations can be used in any 

order, i.e. more than a small number 
of move 'patter ns should be used. 

strategy: a strategy for looking for moves 

should be used, and alternative 
moves should be considered. 

Each issue has procedural information associated with it. 
For each issue there is a function (called the recognizer) 
which determines whether a move exhibits that issue. The 
recognizers are used by the modeller to update the model on 
each turn and by the tutor to determine 1) if any of the 
most recent turns exhibited that issue and 2) whether any of 
the good examples exhibit the issue. Each issue also has an 
evaluation specialist associated with it (the evaluator) 
which can look at a student model and determine whether the 
student is weak in that issue. In addition, each issue also 
has a output generation function (called the speaker) which 
explains the issue to the student, i.e. "I notice that you 
seldom move backwards". 
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FIGURE 2.4 



(RAflKS 



{(1 
(2 
(4 
(7 
(10 

(11 
(16 



3) 
1) 
1) 
1 

1 



^) ) 



QUALITIES (OPTIMAL 3 GOOD 2 FAIR 4 POOR 0 ERROR 0) 
PATTERNS 

(WFFl (OPTIMAL .0 GOOD 0 FAIR 1 POOR 0 ERROR 0 MISSED/DEST/MOVE 2) 
(WFF2 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/KOVE 4) 
{WFF3 (OPTIMAL 1 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/MOVE 4) 
(WFF4 (OPTIMAL 1 GOOD 2 FAIR 3 POOR 0 ERROR 0 MISSED/BEST/MOVE 2) 
{WFF5 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/MOVE 0) 
:WFF6 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/MOVE 0) 
{WFF7 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/MOVE 0) 
{WFF8 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/HOVE 3) 
(WFF9 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/EEST/MOVE 1) 
(WFFlO (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/MOVE 0) 
(WFFll (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/^:OVE 0) 
(WFF12 (OPTIMAL 1 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/MOVE 2) 
(WFF13 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/EEST/MOVE 0) 
(WFF14 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/.MOVE 0) 
(WFF15 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/MOVE 2) 
(WFF16 (OPTIMAL 0 GOOD 0 FAIR 0 POOR 0 ERROR 0 MISSED/BEST/MOVE 0)) 

ORDERS (ORIG (GOOD 4 POOR 4) 
REV (GOOD 0 POOR 0) 
LMS (GOOD 1 POOR 0) 
SML (GOOD 0 POOR 0) 
OTHER (GOOD 0 POOR 0)) 

PARENS (NECESSARY 2 OTHER 0 NONE 7 ERRORS 0) 

DIRECTIONS (FORWARD (GOOD 5 POOR 4 WAS/BEST/MOVE 9) 

BACKWARD (GOOD 0 POOR 0 WAS/BEST/MOVE 0)) 

SPECIALS (TOWN (TOOK 3 WAS/BEST/MOVE 5) 
BUMP (TOOK 0 WAS/BEST/MOVE 1) 
SHORTCUT (TOOK 0 WAS /BEST/MOVE 2)) 

STRATEGIES (SPECIAL 3 MAXDELTA 3 MAXVAL 3 ENDGAME 1 MAXNUMB 1 OTHER 6) 

ARITHMETIC/ERRORS 0 

TOTAL/MOVES 9 

GAMES/PLAYED (WON 0 LOST 1 TIED 1)) 
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Model 



The model maintained by the West system is a record of 
how the student has performed with respect to a particular 
set of issues. It is built incrementally after each move by 
the issue recognizers. Figure 2.M shows Debbie's model at 
the end of the protocol. In this section we shall discuss 
Debbie's model in detail and explain how it is constructed. 

The model is broken down into sub-parts which are 
maintained independently by the recognizers. The major 
parts of the model deal with the general quality of the 
move, the form of the move expression (pattern), the order 
of the spinners, the use of parentheses, the possible 
strategies, and the use of special moves. 

Rank and Quality 

• -Each move that the student makes is judged and given a 
ranking and a quality class. The general criterion for 
judging a move is how it compares to what the 
(mathematically optimal) expert would do in the same 
situation. This expert works by generating all the possible 
moves. Each of the moves is then simulated to find the 
ending positions of both the player and his opponent. For 
the legal moves, the difference between the player's final 
position and his opponent's final position (called the delta 
or difierence) is calculated. For example if a player 
starts at 5 and his opponent at ?5, the delta for a move of 
5 is -5 since the player would finish his turn at 20 (after 
getting a town) while his opponent would remain at 25. The 
legal 'moves are ordered from largest delta to smallest 
delta. The rank of the student's move is its position on 
this list (1 being optimal).* The quality of a move is a 
further classification of the RANK as OPTIMAL, (Rank=1), 
GOOD ^Rank=:2-6), FAIR (Rankr7-20) or POOR (Rank>20).** Both 
the rank and quality of each move are saved in the model. 
This information is used by the tutor to determine the 
general "strength" of a player so that "weak" players are 
not criticized for making GOOD (as opposed to OPTIMAL) 
moves . 



(*)If the same number could have been calculated several 
different ways, all of the possibilities excluding 
commutativi ty of addition and multiplication, are included 
on the list. This has the effect of ranking the moves 
according to the number of ways of getting a better move, 
e^g^ even though 8 was the only number the student could 
have made that was better than the 5 he made, if there are 
six ways of making an 8, the student's move ranks 7th. 

(^^^)The total number of legal moves can vary greatly but it 
is typically on the order of 35. 
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Patterns 

The pattern recognizer deals with the form of a 
student's move expression. A move is classified into one of 
16 possible patterns (WFFn) according to the operations used 
in the expression and the order in which they are performed. 
See Figure 2.5". For example, WFF1 corresponds to taking the 
difference of the sum of two numbers and the third number. 
The mapping between the 16 well-formed expression numbers 
and the specific operations and order of evaluation is given 
in Figure 2.5. Once the WFF number of a move has been 
ascertained, the appropriate pattern counter is incremented . 
The "goodness" of the move determines which quality class 
sub field to increment for that pat tern. This provides a 
profile of the student's use of each pattern. Debbie's 
model (Figure 2.4) indicates that she favors WFF4 (X+Y*Z) . 

Figure 2.5 



WFF Number 


Fo rm 


Needs P 


WFF1 


( A+B)-C 


No 


WFF2 


( A»B)/C 


No 


WFF3 


( A+B)«C 


Yes 


WFF4 


( A»B)+C 


No 


WFF5 


( A+B)/C 


Yes 


WFF6 


(A»B)/C 


No 


WFF7 


A-(B+C) 


Yes 


WFF8 


A/(B«C) 


Yes 


WFF9 


A/(B+C ) 


Yes 


WFF10 


A-(B«C) 


No 


WFF1 1 


A+(B/C) 


No 


WFF12 


A»(B-C ) 


Yes 


WFF13 


A-(B/C) 


No 


WFF14 


( A-B)/C 


Yes 


WFF15 


A/(B-C) 


Yes 


WFF16 


(A/B)-C 


No 



In addition, for those moves in which the student's 
move was not optimal, the MISSED/BEST/MOVE field for all of 
the patterns which would have given an optimal move are 
incremented. This information points out areas where a 
student may be weak and can also be used to avoid 
criticizing the student about issues which were never to his 
advantage to use. 

The pattern section of the model is an example of 
several different issue evaluators using the same 
recognizer. For example the subtraction evaluator knows 
which of the patterns use subtraction and can thus find a 
profile for the student's use of subtraction. Similarly the 
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division evaluator knows which patterns use division. The 
parenthesis evaluator can also use the MISSED/BEST/MOVE 
field of the pattern section to determine if the lack of 
parentheses has affected the student's performance. 

Orders 

The ORDERS field of the model keeps information about 
the order in which the spinners appear in the student's 
move. The order of a move is classified as one of the 
following: (1) ORIG the spinner in the expression occur in 
the same order as they were given; (2) REV, they occur in 
the reverse of the order given; (3) LMS, they occur in 
decreasing order; (4) SML, they occur in increasing_ order 
of (5) OTHER, they occur in some other order. Within each 
class, the moves are kept in two subfields, GOOD and POOR. 
This information is used to make sure the student realizes 
that he may change the order of the spinners. For example, 
Debbie's, model shows that she used the original ordering in 
'8 out of 9 moves. 



Par en the se s 

The PARENS part of the model records the student's use 
of parentheses. Each move is classified as having no 
parentheses (NONE), as having parentheses which were 
NECESSARY as in (1+3)*^ or as having unnecessary parentheses 
(OTHER). In addition, a count is kept of the number of 
parenthesis errors the student makes while trying to form 
her expression. From this information, the evaluator can 
tell if the student understands the purpose of, and feels 
comfortable using, parentheses. 



Di rections 

The DIRECTIONS part of the model records directional 
(forwards or backwards) information about the student's 
moves. Each move is classified as forward or backward and 
the appropriate sub-field (GOOD or POOR) is incremented 
(depending on the quality of the move ) ^he optimal move is 
then classified as forward or backward and the WAS/BEST/MOVE 
subfield of the appropriate field is also incremented.* From 
this the evaluator can determine if moving backwards has 
ever been the best move to make. Notice that in the two 
games Debbie played, moving backwards was never the best 
move. If a situation which exhibits an issue refuses to 
come up, it would be feasible within the tutoring/modeling 

(»)As is the case in several parts of the model, there may 
be several different "best" moves which would classi.y 
differently. In most cases we have opted for picking one of 
the best ones and using that under the combined assumptions 
that (1) it will average out and (2) that field in the model 
is not used for anything critical enough that the difference 
will be significant. 
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paradigm to have the tutor try to set up "interesting" 
situations. For example, the spinner values could be 
(temporarily) dynamically biased to increase their 
likelihood, or hypothetical cases proposed and discussed 
with the student ("What would you have done if..."). 

Spec ial s 

The SPECIALS section of the model monitors the 
student's performance with regard to special moves, i.e. 
towns, bumps and shortcuts. There is a field in the model 
for each type of special move which records both the number 
of times the student made that type of move, and the number 
of times this was in agreement with the optimal strategy. 
It's primary use is to ensure that the student is aware of 
each type of special move. 

Strategies 

The recognizer for the strategy issue keeps track of 
possible strategies that the student might be using. It 
does this by recording for each of the student's moves, the 
strategies under which that move is optimal. At present 
there are five strategies that are recognized; 1) SPECIAL, 
always try to make a town, bump or shortcut 2) MAXDELTA, 
always try to maximize the value of your position minus your 
opponent's, 3) MAXVAL, always try to get the farthest along; 
4) ENDGAME, get to 70; 5) MAXNUMB, always try to make the 
largest number possible. These strategies are not exclusive 
and any particular move may be optimal under several 
different strategies. However, if the student is 
consistently using one of the strategies, it should begin to 
show up. Any move which is not optimal under any of the 
strategies is stored under the field OTHER. As a rough 
approximation, any time OTHER becomes greater than any of 
the strategies, the student is not playing any particular 
strategy.* Debbie made six non-strategic moves, but she made 
three which were optimal under all three of the strategies 
MAXDELTA, SPECIAL and MAXVAL. The fact that these were her 
last three moves would not show up in the model and is one 
reason why the model must be augmented with a history list. 



(*)The picture is complicated by the other issues. As an 
extreme case, if the student doesn't use subtraction 
division or know about towns, bumps and shortcuts and 
doesn't know how to use parentheses, her strategy would 
probably show up as OTHER. This does not mean that the 
student does not have a coherent strategy just that her set 
of possible moves is very limited. For this reason strategy 
should be one of the last issues to be pressed (which 
emphasizes the need for ordering the issues.) 
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other 



In addition to the major sections previously discussed, 
other information is saved in the model. Such information 
includes the total number of moves made by the student, his 
won-loss record, and the number of arithmetic errors he has 
made . 



History Li st 

The history list in West contains a complete temporal 
record of what has occurred in the session. This includes 
for each move, the spinners, the expression entered by the 
student (both parsed and linear forms), the results of the 
move (bumps, towns, etc.) and the final position. In 
addition, a record is maintained of all the errors made by 
the student and the advice given by the tutor. At present 
the history list is used to check the recent moves made by 
the student. This prevents the tutor from "hassling" the 
student about an issue with respect to which he has 
performed satisfactorily in the last (say) three moves. 

Another possible use for the history list deals with 
the problem of "changing the point of view" of the modeller. 
The modeller evaluates a move based on its comparison to an 
expert's move in the same situation. This expert must use 
some strategy to decide which move is best. (For example, 
is it better to get one farther or to be on a town?). 
Whatever strategy the expert uses, (it currently uses the 
maximum delta strategy), it may not be the same strategy 
employed by the student. When this is the case, the 
student's moves won't be evaluated correctly using the 
expert's strategy. Since the reason for tutoring the 
student is not necessarily to teach him our notion of 
strategy, but instead to see that he is aware of the range 
of issues, it might be beneficial to criticize the student 
within his own strategy. If we discover that the student is 
playing a coherent but different strategy (either by asking 
him or by noticing patterns in his model») the modeller can 
try to re-model the student using the history list and an 
expert who plays under the student's strategy. If we are 
correct about the student's strategy, this new model should 
indicate a better player. (At this point, if we verbalize 
this strategy to the student, we can make him aware of it 
and hence willing to consider alternatives. This gives him 
a purpose to the arithmetic practice; i.e. a tool m 
studying strategy.) 



(*)The types of patterns in the model might be a large 
number of moves which are optimal in a strategy, together 
with general strengths in other areas, i.e. when the 
student is making less than optimal moves which can't be 
explained by the issues. 

5 0 
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The Tutor 

In the previous sections we discussed the structure of 
the student model that is constructed. In this section we 
shall see how this model is used by the evaluators to 
determine when to tutor the student. 

Patterns 

The PATTERN evaluator checks the student model to see 
if the student is varying the form of his move* As 
mentioned earlier, the pattern recognizer classifies each 
move as one of 16 patterns depending on the operations; and 
their order of operation. Thus if the student is always 
forming A+B*C, the WFF4 field of the Pattern section will 
have a large portion of the moves. Notice, however, that 
constant use of a single pattern does not necessarily 
indicate that the student is stuck. It may be the case that 
in these particular situations, the student made the best 
move. For this reason, the evaluator looks at the 
non-OPTIMAL subfields of each WFF to determine how often the 
student used a form when it was not the optimal thing to do. 
^The actual algorithm it uses (which is subject to change) to 
determine if the student is stuck in a pattern is "Has the 
student used this pattern non-^opt imally more than 75% of the 
times that he has not used it optimally." 

Orders 

The ORDER evaluator checks to see if the student is 
trying to use the spinners in alternative orders. Again, 
the important factor here is how the student^s behavior 
compares with the expert's, i.e. how many times has the 
student used the same order when he could have done better 
with a different one. Currently, the student is judged weak 
if he has used the original ordering on more FAIR or POOR 
moves than if he has used any other ordering. This also has 
the effect of quickly informing the student in case he is 
unaware that he is allowed to change the order of the 
spinners . 

Parentheses 

The PARENTHESES evaluators check to see if the student 
uses parentheses. For this issue, the student is judged 
weak if he has used parentheses less times than he has not 
used them. This is one area in which the evaluator should 
be extended. The pertinent question here is not "Does the 
student use parentheses", but "Does the student use 
parentheses when they are required". A more complicated 
evaluator could determine this from the MISSED/BEST/MOVE 
subfield of each WFF form in the PATTERN section of the 
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model. The sum of those subfields for the WFF forms which 
require parentheses is the number of times the student has 
missed an optimal move which required parentheses. 

Strateg ies 

The r/.-.r:^c. v evaluator chicks to make sure the student 
is playiti.; -ome type of strategy. The present system will 
only criticize the student when he appears to have no 
coherent strategy. That is, it will not criticize a 
student's strategy, only his lack of one. A student is 
judged to lack a strategy if he has made more moves which 
are not optimal under any one of the recognized strategies 
than moves which were optimal. A more extensive version of 
tne strategy evaluator should be able to deduce precisely 
what strategy the student is using. In addition to the 
first information about the five strategies that the 
recognizer has been explicitly monitoring, the Strategy 
evaluator can use the information in the PATTERN section to 
confirm more complicated strategies. Once the student's 
strategy is discovered, the comments by the tutor can 
contrast the new example 'with the student's present 
strategy. This would lead to much more pertinent and 
correctly individualized comments. In addition, if the 
system knew the student's strategy, this would feedback into 
the model building process by conditioning the recognizers' 
view of a move . 

Directions 

The direction evaluator checks to make sure the student 
is aware that moving backwards is both legal and sometimes 
beneficial. Since it is possible that a student who has 
played two or three games of West has never been in a 
situation which called for a backward move, knowing the 
expert's behavior is essential to avoid unfair criticism. 
Using the expert's behavior (the WAS/BEST/MOVE subfield), 
the student is judged weak only if he has moved backwards 
less than half the number of times it was optimal to do so. 

Specials 

On° of the things which we discovered by conducting 
experiments with early versions of West was that students 
often overlooked towns, shortcuts and bumps when they 
played Prior to this discovery, these moves were grouped 
together under the class SPECIALS. The process we went 
through to increase the complexity of the modelling/tutoring 
system in this area provides some idea of the ease with 
which new issues can be added. The recognizer, for what had 
Drevio'jsly been specials, was extended to keep records by 
the type of special move (for both the student's move and 
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the expert's.) Similarly, the evaluator had to be changed to 
consider towns, bumps and shortcuts separately. But the 
same function which performs the decision (e.g. "Has the 
student done x at least half of the times it was optimal to 
do so?") could be used in all cases. Also the predicates 
which were used by the recognizers to determine if the 
student's move used a town (for example) had to be made 
available to the tutor to filter the list of possible 
examples (better moves). Finally three speakers had to be 
written to print appropriate comments to the student (e.g. 
"You shouldn't be afraid to bump me. I don't get mad," 
etc.) In all the whole conversion took about two hours 
( wi t hi n the already establ i she d framewo rk) . 

The Speakers 

In the WEST system the speakers are very simple. Each 
has three or four possible phrases for each of three or four 
parts of an explanatory paragraph. This implementation has 
the advantages of being easy to build and providing a 
reasonable variety of comments. However, there are problems 
using speakers which are too simple. The main limitation is 
that a speaker is not aware of the context in which it must 
"talk" (i.e. player positions, moves, etc.) and must 
therefore be overly general or risk making inappropriate 
comments . 

Me th o dol ogy an d Ex per ime n tal Da ta 

When we began designing this system we faced 
uncertainty about what should go into a student model and 
how to guide the tutor into making insightful comments at 
relevant, and only relevant, times in the game. Because of 
the total lack of any comprehensive theory for how to 
develop and use these models, the system was designed so 
that it could be easily and drastically modified. That way, 
we could run subjects on the system, observe the systems's 
behavior and the student's reactions, and eventually compare 
the system's behavior to that of human tutors (ourselves). 

Several hundred hours of subjects have now been run 
using the system, culminating in an e xperimen t with 18 
Boston University summer school students (from the School of 
Education). On a day by day basis, over the two months of 
our initial experiments we constantly changed and expanded 
our system, taking into account the flaws that were 
manifested by the results of the prior day's experiments. 
We also did some fine tuning of the strategies of the tutor 
to make it coincide more with our own intuitions. These 
initial experiments included a substantial variety of 
subjects ranging from ten-year-olds (mostly from the 
Montessori School), professionals and subjects closely 
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resernbliriK DOD trainees. 



Initially, the system would only print out the student 
model at the end of each experimental run. However, it 
quickly became clear that incremental changes in the model 
were nearly as important as the end product. Consequently 
the system now "dumps" a complete mode'' of the student every 
four moves . In addition, the system dumps most o'f the 
information computed by the ''expert'^ along with whatever 
tutorial comments were made. This expert-produced 
information provides a substantial tool for helping us 
eval ua te the tutor . 



we decided that 
; age o f tutoring , 
teachers (from Boston University), 
using the system we asked them to fill 
In addition we held a one hour 
the students discussed their reactions 



During the first week in July, after 
our system had reached a fairly competent 
we) ran the 18 student 
After they finished 
out a questionnaire, 
discussion in which 
to our system . 

Que stionn aire 

Of the 18 people involved in the experiment, 12 filled 
out and returned the questionnaire. The following comments 
apply to this s rim pie. 

All but one subject received advice from the tutor. 
The general feeling about the Tutor was quite favorable. 
Nine subjects stated that the Tutor's comments were 
appropriate to what they were doing. Of the two who 
disagreed, one said that the Tutor was offering a strategy 
which he didn't feel he should follow because it would leave 
him "vulnerable to attack". Eight out of 10 students found 
the comments helpful in learning a better way to play the 
'^ame and nine out of IQ felt that the Tutor manifested a 
izood understanding of their weaknesses! One subject 
commented "I misunderstood a rule; the computer picked it 
up in the 2nd game . 

We are quite encouraged by these results. Not only did 
the subjects sense the "intelligence" of the Tutor in 
knowing when to offer appropriate suggestions, but from the 
discussion that followed they seemed to enjoy the Tutor's 
support . 

Two of the questions asked them to verbalize rules that 
they were using. The first asked if they could state the 
rule that, given any 3 numbers (other than 0), gives them 
the mathematical expression which evaluates to the largest 
number possible- Seven subjects stated the rule fairly 
precisely (sum of the two smallest numbers times the 
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largest). The others gave incomplete descriptions like 
"Add, then multiply". and "Sum of 2 numbers times the 3rd 
number." One subject could only express the rule using an 
example; (2+2)*^. From the discussion that followed, it 
appeared most of the subjects could perform this biggest 
number operation, but had difficulty stating exactly what 
they were doing. This further confirms our belief than an 
intelligent monitor who can deduce the rules under which a 
student is operating is superior to one which requires the 
student to verbalize what he is doing. 

In the second question we asked what strategy they used 
when they played WEST. Amazingly, only three subjects were 
able to give completely coherent descriptions. Most seemed 
to be suggesting the following: "I would usually maximize 
my move, but would always try for a town first. I guess I 
had a hierarchy of moves. Distance - If I was behind, or 
far enough ahead of the computer. Towns - if I could get to 
it and jump a town. Shortcuts - if applicable. It really 
depended on my position at the time. Safe towns looked 
mighty good. I always would bump back the computer if 
possible." One subject said she used the strategy "Tried to 
get to 10 's to go from town to town," although her actual 
play did not mimic this verbalized strategy. 

No one had played WEST on PLATO before. In fact no one 
had played WEST at all before, so comparisons with other 
systems couldn't be made. 5 out of 12 felt that the 
computer sometimes cheats, a result Resnick found in her 
studies of children playing WEST. This is especially 
apparent when the student is losing. The student feels that 
the computer is not choosing its spins randomly, but is 
dealing itself bigger numbers. In fact, our version of WEST 
is set up so that the Expert biases its moves when the 
student is falling far behind. It limits itself to, at 
most, 3 on the last spinner when it's ahead. 

Because most of the subjects were elementary school 
teachers, we asked them on the questionnaire and in the 
discussion afterwards, for comments or suggestions about the 
tutoring aspects of this game for their own students- The 
response was enthusiastic. The group felt that their 
students would love playin??; WEST: "Even high school 
students would love the game." It is interesting' to note 
that this was in response to our version of WEST that 
includes tutoring (which all but one received). We had 
wondered whether the tutoring aspects would make WEST lose 
its game appeal. Our fears seemed to be unfounded. 

One sub jec t fel t that " some students would lose sight 
of the math and get cau.':!;ht up in one response set." He 
thought "the Tutor can eliminate this fault if it can detect 
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(and it apparently does) when the same mode is repeated." 

An interesting aside about adult subjects who are 
unfamiliar with computers concerns how free they feel to 
explore the terminal: We labelled one of the keys HINT. We 
did not tell them that they could press it or what would 
happen if they did. Only 2 people pressed it even once. 
It's clear to us that making use of all the facilities of a 
system, at least with adults, -ioes not come without 
explicitly instruction. 



Conclusions 



The overall sense we had from building and 
experimenting with this system is that it is so easy to talk 
about student models and yet it is so fantastically 
difficult to actually construct a system that can grow an 
insightful model of the student and then use this model in a 
sensitive way to tutor the student. The pedagogical value 
of drawing tutorial examples from the student's work seems 
beyond reproach, yet the intelligence the system must have 
to successfully act on its own is considerable. 
Constructing a tutor who is constantly criticizing i? 
relatively straightforward. The point is to ma-ce one that 
only interrupts when a skilled human tutor would and then 
generates a succinct remedial comment. 

We feel that our WEST system provides the beginning of 
a theory of how this can be accomplished. It also provides 
a glimpse of technical issues which must be confronted in 
actually constructing an operational system that can grow 
and use student models in a provocative way. 
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CHAPTER 3 

STRUCTURAL KNOWLEDGE IN TROUBLESHOOTING 
ELECTRONIC CIRCUITS* 

In troduc tion 

This chapter is an investigation into the structural 
knowledge required to troubleshoot electronic circuits. Of 
particular interest will be the ability for a system to 
utilize this knowledge to discuss troubleshooting as it 
takes place. This ability would include debugging a circuit 
and giving a reason why each measurement was taken or just 
giving comments on other suggested measurements. Such a 
system could function as the basis for modelling the 
knowledge of an expert troubl eshooter , for construe' 'ng a 
structural model of a student's reasoning process and state 
of knowledge in electronics, (i.e. by restricting the 
techniques of the expert until his "simulated" behavior 
coincides with the student's), and for having a tutor 
converse with a student 

Natural vs. Unnatural Inference Schemes 

There are many approaches to building a computer based 
troubleshooting system varying from troubleshooting by 
synthesis to formal theorem proving. Some of these methods 
are convenient for enabling the computer to generate 
explanations of its decisions, but with others it is almost 
impossible to generate explanations. An example of a 
successful troubleshooting strategy which cannot generate 
very good explanations is troubleshooting by synthesis. A 
circuit simulator is used by faulting components to see what 
the measurements would be like, and then comparing these 
values to the observations made in the circuit with the 
unknown fault. This generates a list of possible faults 
implied by the measurements taken so far and suggests future 
measurements. The problem with this strategy, as with many 
others, is that con-clusions for troubleshooting are reached 
using unnatural inferencing rules, (i.e. such as in 
SOPHIE). Unless the inferences made by the troubleshooting 
system are reasonably understandable by humans it is not 
useful for purposes of explanation or for modelling a 
student! Unfortunately this constraint makes the 

troub Irs hooter- modelling (program) very complicated. 

The position this chapter takes is to study how people 
rea son in troubleshooting and then to formalize our findings 
by constructing a system to produce troubleshooting 

T*)This chapter is based on a paper presented by Johan 
Dekleer at the Canadian Society for Computational Studies of 
Intelligence Workshop . 



explanations, models of experts and models of particular 
students. This research also provides a formal 

representation of the knowledge and strategies actually used 
to qualitatively understand electronics and to perform 
troubleshooting. As such, it might also serve as a formal 
methodology for determining the kinds of strategies and 
knowledge that should be taught In intermediate level 
trainint^ courses in electronics. 

An Overall Perspective of Troubleshooting 

Electrical Engineering provides a vast amount of 
information about mathematical relations between quantities 
in electronic circuits. In fact, for the kind of circuits 
studied in this chapter, one can calculate the voltage and 
current at any point in the circuit using sufficiently 
complicated mathematics. The use of such complex 
mathematics is never seen in actual situations! Most often 
the only mathematics one uses in circuit troubleshooting and 
understanding is of a very simple type such as in the 
application of Kirchoff's laws. For more complex situations 
it becomes more useful to model only those aspects which are 
interesting, ignoring other aspects. This will of course 
simplify the problem, but on the other hand we must discover 
just what these interesting qualities are and be aware of 
the fact that they ignore certain details (so in certain 
contexts they can behave incorrectly). This type of 
analysis is most useful for studying the behavior of 
collections of (connected) components. We will call such an 
interesting collection a device. A device is a set of 
components or other devices interconnected in a particular 
way to achieve a certain effect. Electronics already has a 
language for describing the' behavior of devices and the 
handling of exceptions. 

There are two' approaches to understanding circuits, the 
quantitative (Kirchoff's laws) and the qualitative (e.g. 
amplifiers). Each provides different information and is 
used in different circumstances. As we shall see later in 
the chapter these two approaches require radically different 
troubleshooting strategies. 

Towards a Structural Theory of Troubleshooting 

The way to obtain new information about the circuit is 
to make a measurement. In troubleshooting, new information 
is provided by coin cidence s. In the most general sense a 
coincidence occurs when a value at one particular point in 
the circuit can be deduced in a number of different ways. 
Such a coincidence provides information about the 
T r 5: i^m pt i 0 n s made in the d ed uc t io ns . A coincidence can occur 
in many different ways; it can be the difference between an 



expected value and a measured value (e.g. expected output 
voltage of the power supply and the actual measured value); 
it can be the difference between a value predicted by Ohm's 
law and a measured value; or it can be the difference 
between an expected value and the value predicted by the 
circuit designer. There are numerous other possibilities. 

In general, a troubleshooting investigation into a 
particular circuit proceeds primarily in two phases. The 
first involves discovering more values such as currents and 
voltages occurring at various points in the circuit, and the 
second involves finding coincidence s. The usefulness of 
coincidences is based on the fact that nothing can be 
discovered about the correctness of the circuit with a 
measurement unless something is known about the value at 
that point of the circuit in the first place. If nothing is 
known about that point, a measurement will say nothing about 
the correctness of the components. One actual measurement 
implies many other values in the circuit. The first phase 
of the investigation involves discovering many such values 
in the circuit, and the second involves making measurements 
at those points for which we know the implied values so that 
.;e can see whether the circuit is acting as it should, or if 
something is wrong. 

We will call such an implication a propagation and the 
discovery of a value for a point we already know a 
propagated value for a coincidenc e . When these two values 
are e^ual we will call such a coincidence a corr oboration 
and when they are different we will call it a conflict . 

Information about the faultedness cf components in the 
circuit can only be gained through coincidences. 
Propagations involve making certain assumptions about the 
circuit rind i.hen predicting values at other points from 
chese. These assumptions can be of many kinds. Some of 
them in-'^clve Just assuming the component itself is working 
correctly*. For example, deriving the current through a 
resistor from the voltage across it. Others require knowing 
something about ^ow tho circuit should work, thus predicting 
what values should be. For example, knowing the transistor 
is acting as a class A amplifier, we can assume it is always 
forward biased. Coincidences between propagated values and 
new measurements provides information about the assumptions 
made in th.- propagation. 

Coincidences between propagated val ues and val ues 
derived from knowing how the circuit should work requires a 
teleolcgi' cal descript^'c of the circuit. As indicated 
earlier^ this chapter does not investigate these latter kinds 
of assumptions. Instead, this chapter investigates 
propagations e;nploying only assumptions about the components 




t herrir>e 1 ves . Although, at first sight, the teleolocrical 
analysis of troubleshooting is the more Interesting, it 
cannot effectively function without being able to propagate 
measurements in the circuit! Also, human troubl eshoo ters 
use less and less teleological information as they narrow 
down to a particular fault, and even in the narrowing down 
process there is a constant switching from using 
teleological values and propagating them in non teleological 
ways. So every theory of troubleshooting must include 
knowledge about local and non tel eol og ic al deductions. 

It may appear that in fact this local kind of circuit 
reasoning is essentially trivial and thus should not be 
investigated. This chapter will show that the issues of local 
nonteleological reasoning are, in fact, very difficult. 
Some of the problems are specific only to electronics. 
Others have a very broad range of application to the 
structure of knowledge. However, if we want to understand 
troubleshooting all these issues have to be attacked, not 
just the more interesting teleological ones. 

Some of the problems arise partially because the 
nonteleological knowledge should interact with the 
teleological knowledge. A particular difficult problem 
which will arise again and again is the question of how far 
to propagate values. Often the propagations will be absurd, 
and only a small amount of teleological knowledge would have 
recognized these situations. Part of the effort of this 
chapter is directed into determining what other kinds of 
knowledge and interaction is required, aside from the 
nonteleological, in order to troubleshoot circuits 
e f fee ti vel y . 

The sections that follow present an evolution of the 
knowledge required. The first sections will present a 
simple theory about local reasoning and troubleshooting. 
Then the problems of the approach will be investigated, and 
some of them answered by a more sophisticated theory. Then 
the deficiencies of the theory and how it must interact with 
more teleological knowledge will be discussed. 

The only constraint we will impose on the proposed 
theory is that the explanations for propagations and 
deductions it makes about the faultedness of components be 
easily understandable. To achieve this there are two 
options. The first is to ignore the explanation problem, 
attack the the troubleshooting problem in any possible way 
and then approach the problem of explanation separately. 
The second approach is to design the inferencing schemes the 
troubleshooter uses to be very close to that which humans 
use or could understand, thus eliminating the explanation 
problem. The former approach is tempting because there 



exist complete troubleshooting strategies which cannot 
generate explanation. An example of such a troubleshooting 
scheme is troubleshooting by synthesis: a circuit simulator 
is used to search for all possible faults that explain the 
current measurements, then a similar search is made to 
identify all useful measurements. Unfortunately, the 
process is extremely time consuming and it is inherently not 
able to produce explanations of any kind. For these 
reasons, as well as interest in the study of reasoning 
processes, this investigation will take the latter approach. 

Simple Local Analysis 

The domain of electronics under consideration will be 
restricted to DC circuits. These are circuits consisting of 
resistors, diodes, zener diodes, capacitors , transistors , 
switches, potentiometers and DC voltage sources. All AC 
effects will be ignored although an analogous type of 
analysis would work for AC circuits. It will be assumed 
that the topology of the circuit does not change so that 
faults such as wiring errors or accidental shorts will not 
be considered as possible faults. 

In this section we will present a simple theory of 
propagation. Initially, only numeric values will be 
propagated. Interacting local experts produce the local 
analysis. Each kind of component has a special expert 
which, from given input conditions on its terminals, 
computes voltages and currents on other terminals. For 
example, the expert for a transistor might, when it sees a 
base emitter voltage of less than .55 volts, infer a zero 
current through the collector. 

In order to give explanations for deductions, a record 
is kept as to which expert made the particular deduction. 
Most propagations make assumptions about the components 
involved in making it, and these are stored on a list along 
with the propagated value. Propagations are represented as: 
(<type> <location> (< local-exper t> <compc ?nt> <arg>) 
<assumption-list>) 

where : 

<type> is VOLTAGE or CURRENT. 

<location> is a pair of nodes for a voltage and a 
terminal for a current. 

The simplest kinds of propagations , require no 
assumptions at all, these are the Kirchoff voltage and 
current laws . 

()2 
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T/1 




T/2 



T/3 



nZ 



n3 



The circuit consists of components such as resistors and 
capacitors etc., term inals of these components are connected 
to node s at which two or more terminals are joined. In the 
above diagram T/1, T/2 and T/3 are terminals and N1, N2 and 
N3 are nodes. Currents are normally associated with 
terminals, and voltages with nodes. 



Kirchof f • 
the terminal 
last terminal 



s current law states that if all 
currents of a component or node 
current can be deduced. 



but one 
is known , 



of 
the 



( CURRENT 
( CURi^ENT 
( CURRENT 



T/1 ) 
T/2 ) 

T/3 (KIRCHOFFI 



N1 ) NIL) 



Since faults in circuit topology are not considered 
KIRCHOFFI makes no nev; assumptions about the circuit. 

Kirchoff's voltage law states that if two voltages are 
known relative to a common point, the voltage between the 
two other nodes can be computed: 



(VOLTAGE (Nl N2)) 
(VOLTAGE (N2 N3)) 
(VOLTAGE (Nl N3) (KIRCHOFFV 



Nl N2 N3) NIL) 



As with KIRCHOFFI, KIRCHOFFV makes no new assumptions 
the circuit. 



about 



One of the most basic types of the circuit elements is 
the resistor. Assuming the resistance of the resistor to be 
correct, the voltage and current can be deduced from each 
other using Ohm's law: 
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R1 



n2 



( CURRENT R 1 ) 

(VOLTAGE (Nl N?) (RESISTORI R1) (RD) 



(VOLTAGE (Nl N2)) 

(CURRENT R1 (RESISTORV R1) (R1)) 

(In all the example propagations presented so far it was 

assumed that the prerequisite values had no assumptions, 

otherwise they would have been included in the final 
assumption list . ) 

These three kinds of propagations suggest a simple 
propagation theory. First, Kirchoff's voltage law can be 
applied to every new voltage discovered in the circuit. 
Then for every node and component in the circuit, Kirchoff's 
current law can be applied. Finally, for every component 
which has a newly discovered current into it or voltage 
across it, its VIC is studied to determine further 
propagations. If this produces any new voltages or currents 
the procedure is repeated. 

This procedure can be easily implemented as a program. 

Strategies need to be developed to avoid making duplicate 

propagations, the basic way to do this is to only consider 

newly discovered values for making new deductions. For each 

component type, an expert can be constructed. We have 
already seen the resistor and Kirchoff's laws experts. A 

uniform interaction between the general propagator and the 
experts can easily be developed. 

The curren' through a capacitor is always zero, so the 
current contribution of a capacitor terminal to a node can 
always be determined. 

(CURRENT C (CAPACITOR C) (O) 

Similarly, the voltage across a closed switch is zero. 

(VOLTAGE (Nl N2) (SWITCH VR ) (VR)) 
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Th<» ^(Mnninini^ kinds of cornporienta are semiconductor 
devices, these devices arc? very different from the 
previously d^ cussed kinds of components. Transistors, 
diodps and zcner diodes have discontinuous regions of 
operation. Semiconductor devices have different regions of 
operation, and each region has a different VIC so that a 
region of operation must be determined before any VIC can be 
used. The transistor has an added complication in that it 
is a three terminal device. 

The diode is the simplest kind of sem "conductor device. 
Basically, the only thing we can say abouL it in our simple 
propagation theory is that if it is back biased, the current 
through it must be zero. 

(CURRENT D (DIODEV) (D) ) 

For the zener diode we can propagate more values. If 
the current through a zener diode is greater than some 
threshold, the voltage across it must be at its breakdown 
voltage. 

(VOLTAGE Z (ZENERI) (Z) ) 

If the voltage across a zener diode is less than its 
breakdown voltage, the current through it must be zero. 

(CURRENT Z ( ZENERV) (Z) ) 

Transistors are the most difficult of all devices to 
deal with. This Is both because it has discontinuous 
characteristics of a semiconductor device and because it is 
a three terminal device. If the current through any of the 
transistor's terminals is known, the current through the 
other terminals can be determined using the beta 
characteristics of the device. Furthermore, if the voltage 
across the base emitter junction is less than some threshold 
(.55 volts for silicon transistors), the current flowing 
through any of its terminals should be zero also. 

(CURRENT C/Q1 (BETA 01 B/QI) (01)) 
(CURRENT C/01 (TRANOFF 01) (Q1)) 



Having experts for each co 
described makes it possible 
throughout the circuits. As 
following circuit fragment: 



ponent type as has been just 
to propagate measurement s 
an example, consider the 
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nit) nl6 n24 n25 

• WA • WV « VA • 

R5 R4 R3 



V D5 



D4 



nl4 



Assume that the fault in this 
breakdown voltage too low 
voltage and voltage across D5 
propagations that can be made 



cirr^-it is that D^l has a 
and t-he measurements of output 

have just been made. The 
are: 



(VOLTAGE (N15 N 1 il ) ) 

(VOLTAGE (N16 N^^' (KIRCHOFFV Nl6 N]>\ N15) 

(CURRENT R5 (RESISTORV R5 ) (R5)) 

(CURRENT D5 (ZENERV D5) (D5)) 



NIL) 



the voltage across the zener D5 is less than its breakdown 



' '" IRRENT 

. : LTAGE 

VOLTAGE 

VOLTAGE 
(CURRENT DH 



Ril 

( N2il 
(N2il 
( N2i| 



N16) 
NU ) 
N15) 



(KIRCHOFFI 
(RESISTORI 
(KIRCHOFFV 
(KIRCHOFFV 



N16) (R5 D5) ) 
Ril ) (Ril R5 D5 ) ) 
N2il N16 NU) (Ril 
N2il N16 N15) (Ril 



R5 
R5 



D5) ) 
D5) ) 



(ZENERV Dil) (Dil Ril R5 D5) ) 



the voltage across the zener D9 is less than its breakdown. 



(CURRENT R3 
(VOLTAGE (N2il N25) 



(KIRCHOFFI N2il) (Dil Ril R5 D5)) 
(RESISTORI R3) (R3 Dil Ril R5 D5)) 
(VOLTAGE (N25 N 1 il ) (KIRCHOFFV N25 N2il NU) (R3 Dil Ril R5 D5)) 

(KIRCHOFFV N25 N2il Nl6) (R3 Dil Ril R5 D5)) 
(KIRCHOFFV N25 N2il N15) (R3 Dil Ril R5 D5)) 



(VOLTAGE (N25 Nl6) 
(VOLTAGE (N25 N15) 



The propagation proceeds one deduction at a time; 
never is it necessary to make two simultaneous assumptions 
in order to get to the next step in the propagation chain. 
The propagation can always go through some intermediate 
step. 

bo 
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A oimpi o Th(.M,)r*y oV Tr'out) 1 er.hoo I. i hk 



This section oxamines how t\\(} profsaj/ 'j L Lori strategy (.'f 
the ppeviou*-; section Crjn be u::;ocj to troubl eshoot the 
circuit. The i dean of conflicts and corrobor'a t ion:^ betwe»Mi 
propap;ation will be used to show how the propagator can be 
used to help in troubleshooting; the circuit. In this simple 
theory we will assume that coincidences only occur l)etween 
propagated values and 'actual measurements. 

The moaning of the coincidencos depends critically on 
tfM? kinds of assumptions that the pi'opagator makes. For the 
coincidences to be of interest every assumption made in the 
derivation must be mentioned, and a violation of any 
assumption about a component must mean that component is 
faulted, Then, when a conflict occurs, one of the 
c::omponents of the derivation must be faulted. Furthermore, 
if the coincidence was a corroboration all the components 
about which assumptions were made are probably unfaulted. 

The usefulness of the coincidence depends critically on 
how niany faults the circuit contains. The usual case is 
that there is only one fault in the circuit. Even the case 
where there is more than one fault in the circuit, the 
approach of initially assuming only a single fault in the 
circuit is pr'obably a good one. 

If there is only one fault in the circuit, all the 
components not mentioned in me derivation of the conflict, 
must be unfaulted. If a coincidence occurs, all the 
:)mpc nen ts used in the derivation can be assumed to be 
unfaulted. In a multiole fault situation these would be 
invalid deductions: in a conflici only one of the faulted 
components need be involved and in a corroboration, two 
faults could cancel out each other to produce a correct 
f i nai val ue . 



If. in the propagation exanple of the previous Srction, 
the vc^tage between N25 and Nl^ was discovered to conflict 
with the propae:ated valine, one of R3, , R^, R5 and D5 must 
be faulted. But, if the values were in corroboration, all 
the components w^uld have been determined to be unfaulted. 

Now t r. a t the f a ' ' ■ t has been e d u c e d to one of R 3 , D ^ , 
R^, R5 and 0^), the propa^ra tions can be used to determine 
what measurement should be taken next. The best sequence of 
measurements to undertake is, of course, the one whic will 
find the faulted component in the fewest number of new 
n e a s u r em e n t s . Assuming that r h .'^ e 1 a t i v e probability of 
which component is fauite ] Ls not , ; )wn, the best strategy 
is a binary search. T h i s i 5 done by examining: all 
n r o p a r^- a t [ o ri s in t h e o i r u 1 t , eliminating: from t he i r 



a s :.ui rn p I i ( ) 1 1 ! i : t, : ; com F)o n o n 1 1\ a 1 r o a cl y d c t o rMn i n e d t o b c 
.-orrect, and pic^kinK a mea:uire?mon t to M)inpi(lc with that 
propagation whoi:^o number of assumption:' is nearest to halT 
the numbtM' of [)ossibly faulted components. 

In the oxaiTiplo there are 5 possibly faulted components, 
henoe th(^ best propap;ations to choose, are those with ?. or 3 
assumptions. That means either measuring the current 
throuf.^! nn, vollaRe across D^l , the voltage across or the 
voltage betwnon N?H and NIS. All the other measur emeu Is , in 
the worst case, can eliminate only one of the possibly 
fa u 1. ted oompon^Mi ts ^^rom consideration . 

Proceeding in this scenario the current through R-'l i'r 
measiir^ed. This coincidence is a corroboration, so R^^ and 
■ire verified to be correct. Therefore one of R3, D'l and R^] 
must be faulted. At this point there are t- -o few possible 
faults to make a binary search necessary. Any measurement 
which would coincide with any propagation having R3, D^l or 
RU as assumptions, but not all three at once, is a good one. 
One such measurement is the current through D^. This 
conflict would Indicate that '^-'i is faulted. 

This kind of circuit analysis can be used for simple 
kinds of troubleshooting^ . Of course, the troubleshooting as 
indi>-ated cannot reallv begin effectively until the first 
conflic" has been found. However, in a more teleological 
framework, teleological assumptions can also be used in the 
propagations. (This transistor is a class A amplifier so 
its base emitter voltage must be about .6 volts.) When 
t^-leoloi^:ical assumptions have to be made, the derivations 
will of course no longer be complete. That is, a conflict 
or corroboration will not necessarily say anything about the 
oom-onents if some teleological assumption was made in the 
propagation. But, as with assumptions about components, 
conflicts and corroborations will still comment on the 
validity of the teleological assumptions in an analogous 
-^•ay. T!^e information provided by a conflict or 
corroboration with a teleological assumption needs a special 
kind of knowledge to make use of it. 
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ected Complexities of the Simple Theory 



The discussion of the previous section presents an 
interesting and, on the surface, a very simple scheme for 
troubleshooting . Unfortunately, the entire approach 1 s 
fraueTht with difficult problems! This section deals with 
sor--' of these problems and attempts to pi-ovide a solution to 
within the orif^inal framework. Such ar investigation 
will ci^irify the deficiencies of using only .ocal circuit 
kn cw : -'d for i; rcjb I eshoot ing . 



Basically, three kinds of problems arise. First, the 
handling of corroborations and conflicts leads to faulty 
assertions in certain situations and thus they should be 
examined much more closely. Second, it will be shown that 
the propagation scheme, the knowledge contained in the 
experts and the troubleshooting strategy are all incomplete. 
All of them cannot make certain kinds of deductions which 
one might expect them to in the framework that has been 
outlined. Finally, accuracy is a problem; all components 
and measurements have an error associated with them (if only 
a truncation or roundoff error) , and these cause many kinds 
of difficulties in the entire strategy. (In the remainder 
of this chapter it will be assumed that the circuit under 
consideration contains only a single fault.) ' 

The nature of corroborations requires closer scrutiny. 
It has already been shown that every c mponent which a 
derivation depends on is in the assumption list of that 
derivation, and so a conflict thus localizes the faulted 
component tc one of those in the mentioned assumption list . 
For corroborations, the simple troubles ho o ting scheme used 
the principle that a coincidence indicated that all of the 
components in the assumption list were cleared from 
suspicion. This principle must be studied with much greater 
scrutiny as there are a number of cases for which this 
principle doesn't hold. 

In order to do this we must examine the precise nature 
of the propagations, and, more importantly, examine the 
relation between a single value used in a propagation with 
the final propagated value. Consider a propagated value 
derived from studying the component D, let the resulting 
current or voltage value be f(D). The propagator is 
entirely linear, so the propagated value at any point can be 
written as a linear expression of sums of products involving 
measured and propagated values. For every component, 
current and voltage vary directly with each other and not 
inversely. Hence, in the expression for the final 
propagated value, f(D) can never appear in the denominator. 
So the final value can be written -'^ : 

val ue = f . ) ) * b + c 

where b and c are arbitrary expressions not involving D. 
The relation between f(D) and the final propagated value is 
characterized by b. By studying the nature of component 
experts, the structure of b can be determined. Every expert 
either multiplies the incoming value (we will denote this 
value which ir. used by the component expert to derive f(D), 
v(D)) by a parameter, or applies a simple less-than or 
greater-than test to the incoming value v(D) to obtain a 
propagated value. Since many components of this type can be 
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involved in a single propagation, each propagation of this 
kind has a predicate associated with it indicating what 
conditions must be true for the propagation to hold. With 
both kinds of propagations, there is a problem if b is zero. 
In that case, f(D) has no influence on the final value and 
so a coincidence indicates nothing about the validity of 
f(D) . 

In the case where a predicate must hold for the 
propagation to be made, a corroboration only indicates that 
the incoming value v(D) is within a certain range, thus 
saying little about the assumptions which were used to 
derive v(D). Note, however, that in a conflict situation 
the predicate is violated, and thus the possibility of V(D) 
being incorrect cannot be ignored. Any single propagation 
makes many assumptions, some of them may involve predicates, 
others may not. In a corroboratory coincidence the only 
assumptions which cannot be substantiated are those which 
were made to determine the v(D) which the component expert 
for D only used in a test, all the remaining assumptions can 
be handled with the usual corroboration scheme. We shall 
call such assumptions, which corroborations do not remove 
from suspicion, the secondar y assumptions of the 
propagation, and the remaining, the primary assumptions. 

The situation for which b is zero can be partially 

characterized. Using tr. same assumption more than once in 

a propagation is relatively rare. In such a single 

assumption propagation b must be a single term, consisting 

of a product of parameters (resistances, betas, etc.) or 

their inverses, and since no circuit parameter is zero, b 
cannot be zero. 

Every occurrence of an assumption about D in a 
propagation introduces another term to b. Each of these 
terms must still be a product of parameters. Unfortunately, 
at this point in our research we cannot give a proof why b=0 
is impossible, but only an appeal to a somewhat heuristic 
argument. Consider the case where b is zero. It has 
already been shown that b is a product of circuit 
parameters, and so is independent of any measurements. That 
means whatever value f(D) has that value, no matter how 
extreme, has absolutely no influence on the final propagated 
value. That seems absurd, so b must never be zero. 

What makes this Uscussion v an argument and not a 
proo list hat: 

(1) Any manipulation on the c.:r'.^iiit to alter the actual 
value f(D) must also shift c and value . (just changing the 
specifications of D results in nothing one in terpr eLat ion 
of the argument is: no matter what specification D^has, in 
this particular propagated value it has no influence;. 
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(2) The idea of b=0 as being absurd is extremely difficult 
to formalize and it is intimately dependent on the exact 
nature of the component experts. * 

In conclusion, it should be noted that we have not been able 
to discover any propagation (in a coherent circuit) for 
which b was zero, and so it seems a workable hypothesis that 
b cannot be zero. Of course, if b is very small, accuracy 
issues become critical, but this will be discussed later. 

The propagation scheme cannot make all the propagations 
that one might reasonably expect. Incompleteness of this 
type manifests itself in two ways; yet, in both certain 
obvious propagations are not made. One is just a problem of 
circuit representation, and the other is an inherent problem 
of the propagator. 

Kirchoff's current law can apply to collections of 
components and nodes, not just single components and nodes. 
Recognizing relevant functional blocks in the topology of 
the circuit is a tedious (yet per form able) task. Circuit 
diagrams usually present a visual organization so that such 
functional blocks (and teleological organization) become 
clear. 

The process of propagation as outlined consists of 
using a newly discovered value to call an expert which can 
use that value to make new discoveries. The called expert 
then looks at the environment, and from this deduces new 
values for the component about which it is an expert. The 
communication with the environment always involves numeric 
values. Experts cannot communicate with each other; 
neither can they handle abstract quantities. Furthermore, 
propagation stops when a coincidence occurs, and iteration 
toward an accurate solution is never attempted. This can 
become a severe limitation in certain feedback situations. 

This entire scheme is motivated by what we see in human 
t roubleshoo ters . The strategy has some very surprising 
limitations. The fact that only one expert is invoked at 
any one time means that only one assumption can be made at 
any step in the propagation process. This means that 
propagations which require two simultaneous assumptions 
cannot be made. Most propagations which require more than 
one assumption do not require simultaneous assumptions since 
they can be derived using some intermediate propagation 
(e.g. all the previously discussed examples). 

One such case requiring simultaneous assumptions is the 
voltage divider. 
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Suppose V and i are known, the current through R1 (and hence 
through R2) can be propagated by simultaneously assuming the 
correctness of both R1 and R2. 

= i1 R1 + 12 R2 
i = i1 - i2 

i1 = (V ~ i R2)/(R1+R2) 

Admittedly, the voltage divider is an important enough 
entity that it should be handled as a special case pattern, 
however, problems of this kind of incompleteness will arise 
in other situations, and it will not be possible to design a 
special case pattern for each of them. 

If multiple faults are allowed, simultaneous 
assumptions must be handled with even greater caution. For 
example, a propagation involving a simultaneous assumption 
can propagate a correct value even though both components 
which the assumptions were about were faulted. In the case 
of a voltage divider, the resistance of both R1 and R2 could 
shift without affecting the voltage at the tap, yet the 
voltage divider would present an erroneous load to the 
voltage source to which it was connected. 

In order to illustrate some other difficulties, 
propagations requiring simultaneous assumptions can be 
characterized differently. If a measurement is made for 
which a propagation can be made to coincide with the 
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orip:inal measurement, a previously incomplete propagation 
has been completed. The coincidence indicates that if the 
propagator could have made an abstract hypothetical 
measurement and used a relaxation or algebraic method, the 
actual value for that point could then have been determined 
without making the measurement n the first place. However, 
since the current propagatio. scheme cannot make such 
hypothetical measurements, a later measurement might play 
the role of generating the hypothetical measurement. 
Unfortunately, the coincidence rarely occurs at the exact 
point of the measurement; all propagations proceed in a 
breadth first direction from the original measurement point, 
and even if this was modified, it would not 'alleviate the 
difficulty because the new measurement might only cause a 
later propagation some distance away from the original 
measurement point which plays the role of a hypothetical 
measurement. The problem then, is that coincidences need 
not be between propagated values and measured values, but 
can also be between two propagatedvalues. 

Conflicts and corroborations between propagated values 
must then be considered. If one of the propagations has no 
unverified assumptions, the coincidence can be handled as if 
it were between a propagated value and an actual 
measurement. However, if both propagations have unverified 
assumptions, the coir:^ i .lence becomes far more difficult to 
analyze. The effects of such coincidences depend critically 
on whether the intersection of the unverified assumptions in 
each propagation is empty or not. First we will examine the 
case in which the intersection is empty. A conflict reduces 
the list of possible faults to the union of the assumptions 
used in the propagations. The corroboration between two 
disjoint propagations indicates that this value is the 
correct one, hence it can be treated as two separate 
corroborations between propagated and measured values. 

The case of a nonempty intersection is the most 
difficult. If the coincidence was a corroboration, a fault 
in the intersection could have caused both propagations to 
be incorrect yet corroborating. Yet, what can be said about 
the nonin ter sec ting assumptions in the propagations? If 
there was a fault in one of the nonin tersec ting primary 
assumptions it must have caused a conflict, so all the 
nonin ter sec ting primary assumptions can be verified to be 
correct. If the coincidence was a conflict, the list of 
possibly faulty components can be reduced to the union of 
the assumptions. In this case it is very tempting to remove 
from suspicion all those components mentioned in the 
intersection, this would capture the notion that correct 
propagations from a single (aloeit incorrect) value must 
a 1 ways corroborate each other or, equivalently, that each 
point in the circuit has only two values associated with it; 
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a correct value and a faulted value. 



In the coun ter*e v^Tipie , an emitter current is propagated 
throup;^ s l»^ansiscor "oo obtain propagated values for the 
base and coll/ctor currents. The base emitter junction of 
thic transistor has shorted and consequently both these 
propagated values will be in conflict with the actual values 
? n the circuit. These two values will also conflict with 
ea jh other. 

Consider the circuit fragment: 



n3 



n4 



R2 



R3 




The circuit is faulted with the base emitter junction of 0 
shorted, with the collector terrr. ir-al open . Thus far the 
voltage at N2 and N5, and a current into N2 has been 
measured. Next the emitter current of Q is measured from 
which the base and collector currents are propagated. The 
initial measurement did not coincide with any propagated 
value, yet a conflict will occur within the propagations 
caused by this measurement. Furthermore, the values of the 
conflicting propagations conflict with the actually measured 
value. The exact point at which this conflict will occur 
depends on internal details of the propagator. Two obvious 
points at which the conflict can occur is at the voltage at 
N3-N^ and the current through the base of the transistor. 

7 4 
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(VOLTAGE (N2 NO)) 
(VOLTAGE (N5 NO)) 
(CURRENT E/Q) 



(CURRENT B/Q (BETA Q E/Q) (Q)) 

(CURRENT C/Q (BETA Q E/Q) (Q)) 

(CURRENT R2 (KIRCHOFF N2) (Q)) 

(CURRENT R3 (KIRCHOFF N5) (Q)) 

(VOLTAGE (N^ N5) (RESISTORI R3) (R3 Q)) 

(VOLTAGE (N3 N2) (RESISTORI R2) (R2 Q)) 

(VOLTAGE (N3 NO) (LOOP N3 N2 NO) (R2 Q)) 

(VOLTAGE (N^ NO) (LOOP N^ N5 NO) (R3 Q)) 

This results in two conflicting voltages at N3-N^ one x:. 
higher than the actual value in the circuit, and'the other 
is 1 ower . 
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All measurements in the circuit and all circuit 
parameters have errors associated with them. Even if we 
assumed perfect measurements, truncation and roundoff errors 
would cause problems. One way to view the problem is to 
study the size of b relative to the error in c. If b is 
smaller than the error in c, a large error in some f(D) 
could go detected. Arain we see the greatest problem lies 
with corroborations. In a corroborating coincidence we must 
make absolutely sure ,.,nat an error in any of the verified 
assumptions could have been detected in the value (i.e. b 
is not too small ) . 

The solution is quite simple; instead of propagating 
numeric values through the circuit, we propagate values and 
their tolerances, or just ranges of values. Each 
measurement and circuit parameter could have a tolerance 
associated with it, and the arithmetic operations cou: d b-^ 
modified to handle ranges instead of numeric values" 
Instead of computing b and its tolerance, the propagator 
could note whenever an error in some incoming value could be 
obscured in larger errors in other values (remember, errors 
in parameters and measurements are usually percentages, and 
thus adding a large value and a small value will often 
obscure an error in the small value). Since such problems 
occur only with addition and subtraction of ranges, 
KIRCHOFFV and KIRCHOFFI are the only experts which need to 
be directly concerned with the accuracy issue. 

Assuming that errors in values are roughly proportional 
to their magnitude, those propagations involved in a sum 
whose magnitude is less than the error in the final result 
should not be verified in a corroboration of the final 
value. (As this assumption is not always true, some 
assumptions may not be verified in a corroboration when they 
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should be.) KIRCHOFFV and KIRCHOfFI can easily check for 
such propagations. Fortunately, a category for assumptions 
which should not be verified in a corroboration has already 
been defined, these are the secondary assumptions. So, 
primary assumptions of the incoming values into a KIRCHOFF 
may become secondary assumptions of the final result. 

As usual, this theory of handling accuracy has subtle 
problems. If the only possible effect of a particular f(D) 
was described in a propagation, then no matter how 
insignificant its contribution was to the final value, a 
coincidence should verify D since it wouldn't matter in such 
a case if D were faulted or not. Furt- '^ermore , the 
propagation through certain components is so .xscontinuous 
that ~ no matter how insignificant its propagatory 
contribution is, a fault in the final value would so greatly 
affect the propagation that the assumption in question 
should really be treated as a major assumption. An example 
of the former is a switch in series with a resistor, and an 
example of the latter is a zener diode contributing zero 
cur r en t to a node . 

Consider the case of a r>?sistor in series with a 
switch. The only contribution of that switch to the circuit 
is in the voltage across the switch and the resistor. A 
voltage across a closed ^witch is zero, so unless the 
resistance of the resistor is zero the swinch becomes a 
secondary assumption of the final voltage. Unfortunately, a 
corroboration with that voltage should indicate the switch 
was acting correctly. 

Similarly, a zener diode contributing zero current to a 
node will always become a secondary assumption of the 
KIRCHOFFI propagation. But, a corroboration should indicate 
that zener was functioning correctly. That is because the 
propagation would not even have been possible if the voltage 
across the zener was near its breakdown. A heuristic 
solution to this problem is not to secondarize propagations 
with zero value which were just propagated from 
discontinuous devices . This , of course , makes the 
teleological assumption that the discontinuous component 
makes a significant contribution whenever it is contributing 
a non zero value, as is almost always the case with the 
switch, diode, zener diode and transistor. 

Accuracy brings along other problems, testing for 
equality between ranges becomes a rather useless concept. A 
simple workable strategy is to use a rough approximation 
measure such as accepting two ranges as equal if the 
corresponding nndpoints of the two ranges are within a 
certain percentage of each other. More satisfactorily, the 
actual width of the range should also enter into 




consi JePcit Ion so that if one end of the range is extremely 
smaii relative to the other, a much more liberal percentage 
is used to compare the smaller endpoints. One certainly 
would want the range [0 , 1] to be roughly equal to [lOE-6 , 
1]. Using the percentage of the endpoint largest in 
magnitude as a fixed range to compare the smaller endpoints 
appears to be the best strategy. A coincidence can be of 
three kinds, the ranges can be approximately equal (or Just 
significantly overlapping) which is a corrobora tion / t he 
ranges can be disjoint which is a conflict and the ranges 
can overlap, but not significantly which provides no 
information at all. 

Havine the the propagator propagating ranges brings up 
the idea of allowing components to individually propagate 
higher and lower limits in the circuit. Every diode could 
propagate a non-negative current through itself. A voltage 
could be propagated at every part of the circuit whose upper 
limit was the magnitude of the sum of all the voltage 
source.- in the circuit. More interestingly, it could handle 
the problem of having a range propagated over a 
discontinuous device: a [-1 , +1] current range into 
diode should have its lower limit modified to 0 (i.e. [0 , 
+1]). Interesting as such new kinds of propagations may be, 
they require separate derivations for the upper and lower 
limits for each range, and thus introduce incredible 
difficulties for handling coincidences. 

When a significant propagation occurs which overlaps a 
test poi*^*-. of a discontinuous cor'.ponent the best strategy is 
to interpret that Tieasurement to have too wide an error 
associated with it and stop the propagation there. In 
general , when '^^rror tolerances in propagated values become 
^.bsurd (a sifi:nificant fraction or multiple of the central 
value) the propagation should be artificially stopped. 

There remain certain characteristics of the devices 
that are not captured in th^ propagation scheme. These are 
usiially the maximum ratings of the components. The base 
emitter voltage of a transistor cannot exceed a certain 
value, the voltage across a capacitor cannot exceed its 
breakdown voltage, the voltap-e across a zener diode cannot 
exceed its break'l-wn voltage, the Dower dissipation in a 
reci.stor cnnnot exceed its vjattage rating, etc. These, in 
fact, can be quite easily captured by simple modifications 
of the component experts. Each expert could check whenever 
it was invoked whether any ratings about the compon.nt were 
exceeded. Such situations of excess can be of two kinds, 
the final value calculated to compare to a rating may or may 
not involve the component itself as an assumption. If the 
component itself was used as an assumption, the situation 
can be treated as a conflict with the calculated rating. 



otherwise, if the component itself was not mentioned in the 
assumptions, the situation must again be handled as a 
conflict, except that the component in question must not be 
removed from suspicion. 

Often a component which is considered possibly faulted 
because of a conflict can really be eliminated from 
suspicion by examining exactly what kind of fault the 
conflict implies the component might have. (i.e. all the 
currents in the transistor must shift so that K:>-choff»s law 
is violated, or, a more trivial case (whic: should be 
handled differently but nevertheless serves a. a good 
example) a capacitor for which the fault of oo low a 
current is entertained). 

In order tr determine the kind of faults a particular 
conflict implies, it must be known whether the value is high 
or low. This can only be determined for conflicts with 
measured values. For conflicts between propagated value 
there is no convenient way of determining the possible 
faults except by hypothesizing all the possible high/low 
combinations and using the intersection of all the results. 

We must tackle the problem about .ow to scan^ back 
through the propagation to determine what faults in the 
components could have caused the final conflict. Of course, 
a straightforward way to do this would be to compute b for 
every component f(D) involved in the propagation. For every 
two terminal component the possible fault can be immediately 
determined from b (unless of course w^- have the inaccurate 
case where the range for a spans zero). The only three 
terminal device, the transistor, requires a more careful 
examination as it has many possible fault modes, and a 
single consideration of a propagation from it may not 
uniquely determine the fault mode. 

Continuing in the spirit of the original propagation 
scheme, a method different from computing b should be used. 
A simple scheme can be derived, which has difficulties only 
in certain kinds of multiple assumption propagations. The 
conflict indicated that the propagation was in error by a 
certain shift in value in a certain direction. This shift 
can be oropagated backwards through all the experts except 
KIRCHOFFI and KIRCHOFFV. The Kirchoff^s laws experts 
involve addition, so each of the original contributors to 
the sum must be examined. For those contributors whose 
(unverified) assumption list does not intersect with any of 
the other assumption list^, the shift can be propagated 
back, after adding the appropriate shift caused by the 
remaining contributors. For those contributors with 
intersecting contribution.^, it must be determined for each 
of the intersecting components whether all contributions of 
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all the possible faults do not act against each other (e.g. 
will a shift in the resistance of the component both 
increase a current contribution to a node and decrease it 
through another path?). For such canceling intersections, 
nothing can be said about the intersecting component. In 
actuality, all this does is capture qualitatively whether 
the signs of the ter- s of b are different and thus 
canceling. It should be noted, that if it really turns out 
to be the case that b can be zero, such a scheme could be 
used at least to eliminate faulty verifications from taking 
place, again at the cost of sometimes not verifying probably 
unfaulted components. 

Incompleteness in the propagation scheme introduces 
incompleteness in the troubleshooting scheme. Even if the 
propagation scheme were complete, the troubleshooting scheme 
would be inomplete. The earlier answer to what is the 
next best measurement is inaccurate. The measurement which 
reduces the list of possible faults by the greatest number 
is not necessarily the best measurement. Future 
measurements must also be taken into consideration, a poor 
first measurement may set the stage for an exceptionally 
good second measurement. 



The choice of best measurement depends of course on 
what is currently known about the circuit. The most general 
approach would be to try every possible sequence of 
hypothetical measurements and choose the first measurement 
of the best sequence as the next measurement. Again, that 
would be an incredible, and unnatural computation task. The 
current troubleshooting scheme does not try to generate all 
possible sequences and only considers making those 
measurements about which it already knows something (so to 
produce a coincidence). 

Since only measurements at points about which something 
is explicitly known are considered, the information provided 
by coincidences between solely propagated values cannot 
enter into consideration. Thus the basic simple paradigm of 
the troubleshooter is to make no hypothetical measurements 
and only those propagations with unverified assumptions. 
Unexpected information, such as that provided by 
coincidences between propap;ated values, is not considered. 

Issues of accuracy are sufficiently captured bv primary 
and secondary assumptions. The binary search for the best 
measurement must of course be reorganized. Since a 
corroboration may eliminate less components from suspicion 
than a conflict could, the search is not purely binary. A 
workable solution is to just take the average of the number 
of components which would be verified in each case as the 
measurements ratinp;. Then that measurement whose rating was 
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nearest to. half the number of faulted components could be 
chosen as the next measurement. 



There remains the issue of generating an explanation 
for this choice. Although the above argument for deriving a 
future choice of measurement could be made understandable to 
humans it does not always generate a very good explanation. 
A large part of the explanation for a future choice of 
measurement involves indicating why a certain component 
cannot be faulted (incomplete understanding by the student). 
Once a component is eliminated from suspicion for any reason 
it is never considered again. However, a later measurement 
might give a considerably better explanation for its 
un faul tedness . The problem of generating good explanations, 
of course, also must take into account a model of the 
student and what he knows about the electronics and the 
particular circuit in question. This is a topic of current 
i nvest igation . 

On the topic of selecting the most comprehensible 
choice from a number of otherwise equally good measurements, 
something can be said. The above scheme for selecting 
measurements does not take into account how "close" the 
mea.' jrement is to the actual components in question. ^ For 
example a voltage measurement across two unverified 
resistors is just as good as a measurement many nodes away 
which also has only those two resistors as unverified 
assumptions. Fortunately these can be easily detected: 
just remove from the list of possible neasurements all those 
which are propagated from other elements on the list. 

The Necessity and Utility of Other Knowledge 

In this section we will attempt to characterize where 
and why local and non tel eol og ic al reasoning fails. Indeed 
many of these failures have already been demonstrated. Our 
method of attack will be from two directions. First, 
inherent problems in the earlier propagation scheme can be 
alleviated with other knowledge about the circuit. Second, 
many of the kinds of troubleshooting strategies we nee in 
humans cannot be captured by even a generalization of the 
proposed scheme. One of the basic issues is that of 
teleology. The more teleological information one has about 
the circuit, the more different the troubleshooting process 
becomes. Currently, most of the ideas presented in this 
paper so far have been implemented in a program so that much 
of the discussions derive their observations from actual 
interactions with the program. 

Obsprvinff the ro pa s; a t ions the oropagator makes , the 
'nest arresting observation is that it cannot propagate 
values very far, and at other times it propagates values 
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beyond the point of absurdity. Examining those propagations 
which go too far, the most dominant characteristic is that 
either the value itself has too high of an error associated 
with it, or that the propagation itself is not relevant to 
the issues in question. The former problem can be more 
easily answered by more stringent controls on the errors in 
propagations. The latter requires an idea of localization 
of inter acti on. This idea of a theater of interactions 
would limit senseless propagation, however, it requires a 
more hierarchical description of the circuit. More on that 
later . 



The idea that every measurement must have a purpose 
points out thr basic problem: our troubleshoo ter cannot 
make intelligen easurements until it has, by accident, 
limited the number of possible faults to a small subset of 
all the components in the circuit. After this discovery has 
been made, it can make fairly intelligent suggestions. 
However, as such a discovery is usually made when the set of 
possible faults is reduced to about five components, it only 
can intelligently troubleshoot in the last few (two or 
three) measurements that are made in ,he circuit. 



Clearly, many more measurements ^.re ^ before this 

discovery and the troubleshooter do anything 

intelligent during this period. It will ohown, however, 
that the propagation scheme and the ideas of corroborations 
and conflicts can be effectively used even during this 
period . 

The only way intelligent measurements can be made 
during this period i? by knowing something about how the 
circuit should be behaving, or Just how it behaves. This 
requires teleological information about the circuit. ^or 
example. Just to know that the circuit is faulted and 
requires troubleshooting requires teleology. In the 
situations where the propagator did not propagate very far, 
the problem usually was that some simple teleological 
assumption could have been made. The voltages and currents 
at many points in the circuit remain relatively constant for 
all instantiations of the circuit, and furthermore many of 
them can be easily deduced (e.g. knowing certain voltage 
anri current sources such as the power supply, knowing 
contributions by certain components to be small, etc.). 
Propagation can then proceed much further. Of course, the 
handling of coincidences requires modifications, and a new 
kind of strategy to deal with ^ ^^ological propagations 
needs to be developed. 

If sufficient teleology about t: ^ circuit is known so 
that the transfer functions of cert: 'n groups of corrponents 
are known, assumptionr. of the form "assuming x is in the 
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correct ntate^' or "assuming x is working correctly" can be 
Tiade. Issues dealing with structuring such a hierarchical 
and teleological description are being investigated <Brown & 
Sussman 7^>. 

The pr- pagation scheme of the previous sections can be 
used to understand the implications of these assumptions by 
propagating them in the circuit, and to determine all the 
isomorphisms of a particular set of measurements so that the 
appropriate values for the teleological description 
mechanisms can be discov >red no matter what T.easur ement s are 
made . 

However, as indicated earlier, a new procedure has to 
be made to handle coincidences. At a low level, 
coincidences can be used quite simply. When it is 
discovered that a certain voltage is lower than it should 
be, a search can be made in the topology of the circuit in 
order to locate faulty components which might have caused 
such a shift. This would work most of the time, except in 
cases where complex feedback paths were present. 
Coincidences and corroborations involving assumptions 
concerning collections of components need to be handled 
differently. If an entire collection of components is 
working correctly, all the components inside of it can be 
assumed to be working correctly. But, if a collection of 
components is possibly working incorrectly, a measurement 
must be made within the module which can best determine what 
could be wronR. Whi;^ tr^ previous deduction required 
extrinsic knowledge ut : nodule, the search for such a 
fault within a module ' quires an intrinsic description of 
i t . 

r "ipr-- ing for reasons why a certain value is not 
i y \-:hal it should l^e , it is important to note 
oxnmin^ng the behavior of a particular component 
ohat the reason for its apparent faulty behavior 
^ther ' .th itself, or what it is delivering values 
is jpplyirg values to it. 



When 
teleolop:i 
tnat wher. 
0 r m o d u " 
can lie 
to , or w' 



Ful ure Re .:ear 



Al th i^'b the discussion of the previous section 
sketches it the necessity for teleology, more basic 
research has to be done into the nature of teleology and 
more work has to be done on the local analysis to even use 
the little we do know about teleology. 

Once a oropagation has been made, it is currently 

impossible to remove it. This raises many problems. For 

■,le , 1 r teleolop;y told you that a voltage at a 

Doint wan 10 volts and you measured it and it 



e xam p . 
particular 
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turned out to be 20 volts the strategy would be stuck. 
There is no ability to forget the original 10 volts and its 
propagations. In our simple single fault theory this is not 
fatal; a clear conflict occurred, and the single fault has 
been found teleology! Similar problems occur in purely 
local analysis. Suppose a zener-diode is off, then we could 
propagate a voltage less than its breakdown voltage across 
it. Unfortunately, if at some later time we would measure 
the actual voltage we would only be able to tell if the 
diodes breakdown voltage was perhaps too high. The original 
propagation could not be forgotten. 

The necessity is for another assumption type which, 
when conflicted with, will merely cauise another assunption 
to be chosen instead of concluding faultedness. This would 
help both the problem of int-racting with teleology and of 
repropag? ting better values. Sussman & Stallman are using 
this idea for circuit analysis. Arbitrary assumptions are 
made about-the the states jf the devices in the circuit and 
when conflicts occu/ different states are chosen. This 
proceeds until a consistent assignment of states has beon 
found. The current troubleshooting strategy could oe 
extended triis way. It is probably a poor idea to choose 
•States arbitrarily, since that would be extremely time 
consuming. A state should be chosen only if some reason has 
been discovered for it. This strategy is workable for 
troubleshooting because many more values are known about the 
circuit than in circuit analysis where the point is to 
discover ^ these values. In troubleshooting, extra faulted 
states should be modele,. so that when a contradiction occurs 
components should be for^ced into different faulted states. 
This would be dene by forgetting the VIC of the old ^tate 
and assuming the VIC for the faulted state. 

Forgetting a propagation is in general nontrivial. If 
a pi^opagation is forgotten, all the propagat: jns that ensuer>* 
from it have to be forgotten also. Unfortunately, 
forgetting this propagation is simple compared to the other 
forgetting that has to be done. Namely, all inferences 
about the circuit, not just those directly about 
measurements, have to reconsidered. For example, 
coincidences and their implications on the faultedness of 
components have to be modified. Most difficult of all, 
deleted propagations may have blocked the further 
propagation of ^ther* values (for example in coincidences). 
These blocked propagations must now be allowed to proceed. 

Forgetting will have to be implemented by storing the 
assumptions c every deduction in a data ':-:iSe so that 
whenever an assumption is deleted all deductions depending 
on that assumption be deleted also. The simple solution of 
a CCNNIVER context mechanism is not useful in this 
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apolic i because the context mechanism can only be used 
effectivel;; if one knows, a priori, what things might need 
be ^rr^otten and that a total ordering for these 
assump^. ^ is known. Both of these conditions are 

imposr,. Ibie to meet. 

Wit^^ such an ability to forget, more versatile local 
propagations can made, multiple faults can be handled and 
teleologv h mu^t eventually be added can begin to be 

hand led . 
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CHAPTER H 

T0.7ARDr) il^ACllINl] MATHHMATICAL PHOBLEM SOLVING 



! n tr^oti uc t i 

There are two i ri t er' t. wi n e d long term gorJ s that 
motivated the research reported in this chapter. The first 
goal is to establish the theoretical foundations and a 
procedural knowlndge u.^se for an "intelligent" generative 
tutorial system for mathematical literacy*. The second is 
to p;ain insight into how to build better information 
prooe3,s Lru:^ models of tutors for this and other domains of 
knowlec' ' e . 

< 

Before proceeding with the technical discussion, 
include helow a hypothetical protocol of a student using the 
kind of tutorial system we are working toward. The protocol 
will pr^ovide a glimpse of the kinds of tutoring we expect 
our final system to be able to provide by using its own 
built-in reasonin^^ strategies. At the outset we stress that 
constructing a .nachine to solve algebra problems is 
rel-^tiv- V straightforward, as i' is to construct a system 
to o:ir-ok ; [le validity of each step of a proposed solution. 
The c na 1. 1 t-nge , howevei', is to exploit these techniques to 
e:r^l:Ie Instructional system to make sense out of what a 
student is doing. That is, the system mus' be able to 
':: alyze his solution path:^. sufficiently deeply that detected 
er^rors can i)0 .rjccinctly and meaningful described, 
iODropriate hints can be given, and automatic remediation 
anc' problem generation can be performed. These requirements 
reflect the subtleties of the kind of tutoring which is 
possible when the tutor can nduce a model of how the 
st.der:*- is reasoning, and what his underlying "bugs"' or 
;)oints c; f ':^on fusion are. 

This md the next chapter will describe our efforts, 
thus far, in und er: '.and ing , iesiPining and implementing such 
a : ystem . 

P =i sic Ass pt 1 0 ns 11 nd e r 1 y i n t n i. Research 

One f ':^'ur basic -assumptions is t'''at a major 
l)reakt hrouRh in the effectiveness of CAI wili. be achieved ' ' 
const ru'-; in? syste^^ th' -^an' follow all of the 

i n termed i- i te steps that - student is performing while 
solving a Riv \n 'V<erci'^e. F- .■ having sufficient built-in 
generative " in to 1 1 i ^eno e " , the system can constrjct an 



(*) .ath'-^ma tical li eracy scDCt-^ms the procedural skills of 
ma*nematics as oppos--,i to its axiomatic theory, (e.g. the 
kir'i of knowledg-^' needed to enable students to comfortably 
ma- '-^ulate ind understand the mathematical formulas 
eri'. ^untered L n basic counS'Vs in mechanics, navigation, 
fli-ht ovnar ?s. ei eo t ro:M. ?s . ^ 
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PROTOCOL 



WELCOME TO THE COMPUTER BASED ALGEBRA WORKSHOP! 
WHAT IS YOUR NAME? 

>> max 

AS I RECALL, WE WERE WO. xING ON REDUCTIONS INVOLVING COMPLEX FRACTIONS. 
YOU WERE HAVING A BIT OF DIFFICULTY DECIDING WHEN TC; MULTIPLY OUT A 
PRODUCT, OR FACTOR IT FURTHER. ARE YOU READY TO TR^ A NEW PROBLEM? 

>>yes 

OK, THIS ONE IS A LITTLE HARDER THAN THE LAST ONE. 

The system yireds to enr " nj a mctHa for the problem complexity which ^ 
vaflri'U^ not only Qone absolute notions, but oXso a knowledge of which 
y*Ui\u> :nd heuristics he student has mastered, 

PLEASE REDUCE; 

YOU ^:an write on the tablht now. 



THAT'S A GOOD START. YOU HAVE CHAN ' ONE OF l; iVIS J ."INS TO A MULTIPLI- 

CATION, AND THE OTHER TO FRACTION L J^J^iaT, YOU AI.GC GOT ^7T., OF THE BRACES, 
WHICH WEREN'T REALLY NECESSAK'^. ?LEASE CONTINUE (BUT DON'T , ORGET 
THE 1/2) ! 

Occasicpially the system comn'nt even on correct wui'k, ensuHng th^it 
the sequence of sterns which were ::jr formed is madr ex^jlicit. fJotic-i 
that a system which consta7'ily tc Id the student^ ''wou must also jopy thr 
i^ould bi^ -c '^7 frustr'ating ^ ' 



Note that the systerr. :ust be auar ' of common notat'i-nnal vari^ations a 
ambi^ -dtics, such as ^ower/upp-^r case for lit^^ral^'. 

(2^- ^ b)x ^ ^ ^ ^_ 

Rejo-tnizing 'h.it only rhe "atte). led'' parts of e:vrussto:i have oeen 
written, while unchangiyig parts are not •■^:opi3-l ..n.t .1 Ijter, can r^^iqutre 
cow' livable vvoblcm solving. 

I AM AFRAID THAT'S NOT QUITE RT':., V; MAX. DC YOU V.HEI-E ^O'J HAVt^ :4AUb A 

SLIGHT MISTAKE? 

Underst andin g an ^^rror means isolating its pvecicr Iccation in th^' o.r^ression 
(and'wJien it occ^''-^:d '>:. tho rul^'-path ) , categorizing it as one of j clas.; oj 
common er.^or typcr , and rel ■ ' 'ig it t^ the student model, 

7> no 



PKHIlAi'j; II' VOU LOOK AT THK LTTJF: ABO.I::. on ThAT i/INE, YOU WERIC MULTIPLY (K' 
OUT IN TIIK DjlNOMINATOR. NOW IN THIS LIN]-, YOU HAVE TRIED TO I'ACTOI'i I"^ 
AGAIN, RIOHTV 

int-'^^'ucti) f.li ^ij-nt-^cncvatyiK ^;/^)^ the ntude>its are Ih-lpeu-' 

» yes 

yiiitiplliiJi:; r- to)> ' r/'i ,\ypvession is a vai mat form 

' ' ••■ '''' •' in l.lui npoblem-i^olviyig st2-ate.,'ieij ^ 

rath. th,iK, .•:..;'/ /(■j>: ';, :.}arf.inular operationa l errors, 

\W''V INSTKAD OF HAVING THE ".L*^" A THL RIGHT, THIS TIME YOU BROUGHT IT OUT 
Ti' THF Ll'^FT. DOIlS IT MAT'I'KR ..m-'\T ORDEl: YOU DO THK MUI/I'IPLYING? 

.'.-^ is thi' louii'vlij i.ti^, priyi-^iplr Jur.tifies the claim that the 

ipl^'-factO!' .wr///. h'lr, rioyi-produotivc . 

>"> o { c;(.>urMO not ! 

PL'T THF.N? 

^ L qiKJss i l(jft: out: a 2 in [2^- Vu) . 

'•";00n! LFT'S ^NTINUF WITH YOUR NEW PROBLEM, AS IF W:- HADN'T NOTICED THE 
MISTAKE, AND ..EE WHAT' DEVELOPS. 

^..loit ^'i:ii:t lra!*}i to j\:':fo^;ni :\r mistakes ^ :nf developing higher-order 
rulf.:r to hid: '7/; .-^lir ^k ^ vo>'/r, ^tii also to sense when something seems 
I^eih-i.-vdup -kd ['is r(-:-:ruircs letting him sometimes proceed, 

dii:' ri:s , rr rui,- .d.mjdi^ ved.efined the problem. Since our 
^lur. ^^10: rsr^Vidin^d' , it can simply solve this new prohl- 

Sr:e id in;- f.hi ng r)f poLt'}itial interest Ln letting him proceed 

li. :yi' !'^:j'-:r-''ig to i nod^.i ^jf his ary^ent sdk.: lls, the system 
d..'': ■':-dd 'ir sLui ^! to l..>r ..: h-i tr Jianid'.c the ncw pvoi^irri 

WHAT \\ MLD YOr: HAVE WAM^'- '^'O DO rJE>:T? 

a 



dw. . ; ;/ ; ; t e m r ; ; /, . y^^; f:(i }ij ( -^a nee 1 1 a t i o r: . 



' ,\s'> ■ -i): ' h \viiddi thr: :r:m;- 
ii j^:.aatd r'}is '.() Oic s:.udenf^ 



» 



3.YV(a- 3b) 




;0 AliKAD FROM THi:.'- 




3 b-l- I 



•J. , ■■' t ll'i:' Ir ! ;a at; or :■;'>■'•■ 
., H') th i>l'j '.■ .7.7 ;' • . ; i, j'i : . 



"> b . ■*- I 
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:"t:t * . -rfiW liur> '< '^' r:> ■ :', •.; .(/. 

Wl.-;hL, Y(M) C'KKTAINi, ' riNISIlI-D THAT ONF. OFF FASTI UNFORTUNATELY, YOUR 
WORK IS NOT QIJITK Kl'. '. THKRF WAS A LITTLE ERROR UP ABOVE, BVT THIS 
LINK HAS A Bir: MISTAKl , WHICH LOTS OF PEOPLE GET COtivrJSED iOUT. DO 

YOl) SEE rr? 

>>woll, i. wasD't sur^' if iAm ciLMiom.inal:or should be (a-O)^a; or (a-1). 
r':uir}:t.: .^mirnofilii ^jowimU: '))i^cudo-rniLer.'' to luuidle unusual (or fo}\iotL">i) 
.d luution;':, lOhitili a>'/'.).:' i>ri*aur. of other ernwo. 

YOU •hi: on THE RIGFiT Tf<ACK . WHAT IF THE PROBLEM HAD BEEN: 

'LD THE ANSWER HAVE BEEN , OR ~- - UNDEFINED? 

» i t". would bo 1 . 

GOOD NOW SUPPOSE YOU WERE TO EVALU E THE FIRST EXPRESSION IN THAT LAST 

LINE, BEFORr CANCELLING 3b, WHEN a-1 AND b=2 . WHAT WOULD YOU GET? 

'^yin.>U'io jui^tiSicn f ion for correijinrss of algebraic ruler^ is part of 
irur.]' iUjimnti^'lT] ftur"L-. icralii in .'■jebraio fovrm.ilas reprr'Scnt numl'crr.. 

ruLca arr' cxp- 'tc^J to prciu:rvr dquivaL'ncr over all ..r,nigrmentc of 
'I'^purri :^al ih'ilui'u: to litoraln. 



I o 



WHAT ABOUT THE SECOND, ~, WHEN a-1? 

a 

m.' ^ ' :.*-i!{f7or, fri ^'hoor.ijin it.r. nwriri'ical i-^xamjlos^ 
^■ i:^rd.: ''0}jfii:':Kj ^ /-r; Z o^ luali t f a' . 

J 

SO THE FTKGT TEF^M IS NOT THE SAME AS THE SECOND, RIGHT? WOULD 



a-] 1-1 |0j 



ANY B''TTER? 



jyirnrrP i) \ i i^rie fo' to ■Z>i I't^.'or^' • 

roi}it. '>:■': o:- rl-. correct role of tly^ t.^ ■■ud., 



» no 

SO WHAT i^OEf FHAT IMPLY? 



a . role in findir.^: the wid(irl-d}Uj 



>> it can't- b.^ eithor of n. i am bcqin;iing to Lnink t\- soin. thinq (jlse 
>> i ^. wrouq . 



TIlEKi;: rs. YOU CAN'T CANCEL 0' T THE ' /" s UNLESS THl'V ARE r WION I 'ACTORS OF 

EVERY VKiv TN Tli:- Ni!MEr<ATOR AN:; DENOMINATOR. S INGE THAT IS NOT THE 

CASE HERE, V )C HAVE CHANGED THE 'AEUE OE "liE EXPRESSION. 

[y.:-.. : : y.:ui.-le:- -or r.- -.>h-. .n. iitu!r>t'r "bug'' 



GOOD. NOW LET'S BACK UP A BIT. HKKE IS WHAT YOU WROTE BEFORE: 



3 ^ I 



SINCE IT V;AS .ILLEGAL TO CANCEL T-IE "3b'" s, WAS TFIERE ANYTHING ELSE TO 
BE DONE? 

['I lurjtvii' 'm; the }*el-'itio}ii.;hip of the error to the f articular contex! in 
ton L eh it occured. 



^ i don't think so. 

p^;ht. so the answer would be 



PERHAPS YOU FELT THAT IT WASN'T 



SIMPLE END" H TO BE THE FINAL ANSWER. THIS MAY HAVE ENCOURAGED YOU TO 
RESORT TO "SHADY TACTICS"! BUT WHEN THE ANSWER ISN'T SIMPLE, IT MAY BE 
BECAUSE OF AN ERROR YOU MADE BEFORE ~- OR IT MAY JUST BE THAT SOME 
PROBLEMS HAVE "MESSY" ANSWERS. WHICH DO YOU THINK IS THE CASE HERE? 

A>i.! draioiruj a (jenex'al lesson aboi^^ ^-^roblem .solving heuristics, 

V>woll, since we changed tho probl*jm way back in the beginning, it could 
>> li.ive a mcGsy ans' "^r. 



THAT'S TRUE. 
FROM: 

3b , 



WE rei);':fined the problem. 



BUT THERE IS ANOTHER ERROR — GOING 



TO 



iil;^- sb ) 



.SINCE YOU (JSU.-\LLY DON'T MAKE TFIIS SORT OF ERROR.. I'LL JUST TELL YC J WHAT 
IT IS: YOU FORGOT TO MULTIPLY THE NUMERATOR OF THE 1/2 BY THE C r.^.ZTO^R, 
(a-'3b) , WHEN YOL: COMBINED THE TWO FRACTIONS. DO YOU SE: WHAT I - : .AYING: 

''■Vw - a ordi I teachiru;! L)trat'::gij too! :\spociallij when -^J e 

ii : 'oul: - '-e --..'t fiouiunv-^'ital . But the descrii lion mu[\t be in t ''--^s vhat 
" " 'y:d under:) tand! 



»ycr. 

SO W?IAT SHOULD COME AFTER 



>> 



3b 



3b 
3b) 



± ? 



)K, GO ON. , . 



bb t Sb) 
3 b i- - '5 b 



THAT'S FINE, 



i^fa.^ ^b) 



iN-ufi»M (iK^-lfM of 3t(^pr> which the stuHont nrTforrripd 
( i rn^ ^ I i ni'- i rn p 1 i ^m. t opera t iorK> ) , and then idcMitlfy and 
.•fitique th'- fun damontrjl errors uhat were committed, ar. 
op[^<^--^^d to simply evaluatiru^ whether or not the Ml/lW^llC 
r r ' t . 

It, is also a.ssumed tliat the effectiveness .)f a tut, or 
can ho f^reatly improved by constructing a detailed nno^del of 
the ^Mirrent state of eac h student's entire set of problem 
solvinfT procedures for the domain, and by taking that model 
into ■.'onsidora t ion when formulating tutorial comments 
apnr<-)f>r iate to the student ^s problem solving behavior. 

Another assumption is that the effectiveness of 
pai'tlcular remedial strategies can be significantly 
i fii-^r'oased by constructing and using a detailed exf)lanation 
ol" he nature of th'-.' error which occurred. T.Us means that 
TM.'M particular error should be seen by the tutor as an 
i-:'t.'i;:ce of a category of rc^ ited error types , a:Mi r h- 
orvMjrrenco of each class of error\s would be seen in turvi r 
indi'^ative of specific weaknesses in the stud-nit's 
UM !'^' i.aniing. Tutorial assistance is to be keyed to Lh^'.'^o 
I -"'c i I'lc weaknesses . 

\ final assumption, implicit in our Foa^s, is that it 
nossibl .' an i . ■-^"■l^nes desirahi.e to instruct stude-;:s Ln 
I :-'r ■■^c-edur'^ a I , ' tv . do it" type of knowledge of ; rjimain, 
•vPhout, bel a/r-f' hig the formal theory underlvlng that d-'-iain. 
- ..^ 1 1: 1 , v;e roqu're thai' our system understand and 
,>v[Lain algebra from tne point of viev., not of the abstra'-:t 
;o£i_ca !. structure of the knowledge, but of a .reorganized, 
l£ilCnLC L-:i£ated structuring of ^^ov; ho is to use the 
kr.^wl'-nr^' :'-'^r soLvinp: algebra orot ,,is. 



] 1 a t I ■') r:: o f h^' se a s sumpt l ^.mi s 

.\r. i-n nr>d ^a " e i'^plication of our appr*oacf' if it is l-^ 

ur^'M^uI -s a orac^-ioal educational teeiiiiology -- is thiat 
■reatiar: i a detailed r e pr so n t a t i o n of the path of 
. prr: v-i i V V' st^-'ps whicn th'^ student mu.at iiave trace--' -.ut , 
ver; 1: i r^^^^vious stan-'^. \iis urrent input, and prior 
w 1 i - ^- r n h e t u i tv ■ , 'in A a 1 r e b r a in --^ a e r a 1 ) , w j ] ■ :i e d 
bp ! ■ ne f*f i c i 'Mit I y . /^irthermore , i t must be 
:n* a t 1 ana i 1 V feasibl-^ to jonstruct a ^h r-ough, 
M , . abl o ^n.-^del of the stud'?nt's o v er . aowl^-'e of 
ebr'Tia~" p- /a^dnres. This can be aohiev' . abstract intr 
rn i n 'i \ v i. :i u '\ 1 o r o I) 1 ^ s o 1 v i n r o t o o 1 s , ' ' o ni .i u n c 1 1 o n 

''r].-)r> '/>:or^ ^tations barsed upon t..^ ■st:em'a own 

jr^^- -isd a^i an i -^rstnnding of oofnmon or lyp".-. 



M'UM'DV'M', i.i) (;rcic»r^ I'im' he tutoring;'; :>y3t(nn t:o tn^ wi.'ioly 
.'ipplioable to stu^ietit,r> wlLh (i i fferent t)ac k^^'oun (i.v^ , Uio 
tax onomy of er r oivs inn. o f I e o t unl vo rsal k i n d r 

in i si'ndor r.tand ings , rath(M' t.ian artifacts of partL^.;L ..r 
t p a h i n i-^ t. y 1 e s o r 1 1-? c hn i q 1 1 p r . 



re pre sen t 
h Lunan- 1 ike 
f?: e n e r a t i rui; 
10 1 work o r 
roor,fi:an i zed 



xpe r t sy 3 1 em 
a n d p r o 0 e 3 s 
way . At each 
expl anatio ns 
pro cedures , 
knowledge . 



for algebraic manipulation must 
algebraic entities in a natural, 
:>top, the system must be capable of 
for its actions, in terms of a 
which embody the pedf; gically 



'Vy n t he tic Appr oao n 

Oii ^ approach to eonstr uc t rig such a systei.'i has taken 
t,wo forms: :.:vnthetic and analytic. The synthetic approach 
concerns the design and partial implementation of a 
collection of modules (programmed in LISP) to analyze and 
critique a student 's solution pa.h. We have run a limiteii 
so^ of exper imencs- on the operational portions of tdie:" 
modules. The purp.:-^ of these experiments is to predl t the 
computational behavior of the modules with respect '.o 
dif:erent formulations o; roduction rules <Newell 72> wh. : ch 
encode chunkr> of alt^v^braiC know -how . In particular we h .ve 
been concerned with und I'standinr the combinatorics of the 
search space involved in fi in the missing steps in a 

r.tudent's pro[ .'^e'' ^n, a::d in characterizing exactly 

what is wrunr wi ' h a ;tud ;nt's answer. Such an 
under, .andinr 1:^ ::rucia! in d e t er::; : n in the extent to which 
brute- forcc^ r-eai^v^h r^trategies ove:' the set of production 
r' 1 1 s rn u s t b p i; ^ p 1 a n t e d by m o r .sophisticated heuristic 
V.' -i 1 cues . 

As the rU.ud-rit becomes rr.jre proficient, the 
•'cn: in ator i a ! nat':;^e of the search space seems to warrant 
>'^v;o:': i; h^M: r i s 1 1 c '■^ . sincp students perform an 
crea5:l:;<- numb'-r f ai.^ebraic mnnipulations "in tht - 
ads* Wo ha;:* already p;iven a '-^cod deal f th'^ught to 
;l irtr t/fv-^se problenis, yr.i^l a f'ew of oui ioeas seem 
orM,;mis i . A Mrtincvion betw^-^r-^ strategies (higher ord'jf 
r^iles whi^h 1 e tt:^ r'T* i s e t'.-^ o r d e of (rasic) ru:- 
1 spl icat ions • , nr. \ the set ^ ^" ( prim i live and ''-^rro) 
operations whi ^h tn- :'tud':nt \- known to be using, gr^ea'ly 
sons^trains the S'/'-^ch :-^pac':^. Wiunout it, any sequence of 
■••^o-M'a t i ^ns won': : n po .-^ s i b 1 i t y . Fur t ;^ermore , rules ire 
;-. j rp np-h i ca : I y s t- ;s Lur '-i - one need not expand sub- rules 
:;-,ti- tr. ■ vop lev-"'-l '^ol ^ i"'n p 1 a n has been discovered. This 
; s \'i.:nilar to tih- sse o MA::R0P3 in the ARSTRIPS system 
■"/ss'erdoti . s-uristi ■ equivalence checking, based upoi. 

] j ; . s 1 v' i ' . t f 1 ^ i o s b 0 d by Mar'^in <MartLn 71 >, 



Ti o e 



s M 



('ur* t. hr r' ( • on :U r m i n : I he searo h by eliminating^ r.(^r) i (\ o ra tj o ri 
o error' o po r\i t i ' mi . 



1 1 o f P ) d u 0 t;. i<^r) H u 1 o j;^ 

In !• ■'t.rcv'^-pt^iM , "r mo oV Iho d(vsign decisions which wero 
made in oun preliminary efforts seem somewtiat 
urisatLsf\'i ^tor'y . Kor example, the organization of the oy.stem 
around '^latively simple rewrite or production rules may 
have be(^n unwise. A more general form of production rule, 
or "scher'ia" now seems prefer able. Such a ^hema might take 
a f o rrn :•• u- : h a s : 

[pattern + conditions => action => new [pattern] 
in which the rlf^ht hand side can represent an entire class 
of expressions such as might be generated by one of the 

ornifnon error categories. Moreover, there should be a 
pf .^vision for [lighei- order, or "meta-rules" for recognizing 
transformation:'. On the basis of introspection, such rules 
arc fVequently er ployed by skilled tutors. Examples might 
be something like the following*: 

(a) IF MA'^CH < in i tial-ex press Lon> Wiril 

(' f ! n *^SinNP //2(QMUMBRRP //^f'JoiGNP N(JMB['R P ) 
ANi' MATCH '■inaU.xpresslo.ii;' WTT.i 

( ! rif^SIGNP //O0NUMBERP) , 
T-'!-;] conclude that RULE was 

"Add i tion-0 f -Two-Si gned-Numbers" 
1 t h t he f o 1 1 owi ng 

AVKAT: IF //6 is no' in the appropriate relation t^.) 

and tn^n prop'/'Se t.iiat studont ^^rr'or occured. 

■: [- rr match <initial-expression> WITH 
( ! // 1 ^'AXPRP ! //.'?AXPRP) 

AND MATCH < f i n a 1 - e x or es s i o n > WITH 

// ^^L I T A TOM P ! // $ ^5 A XP R P ) 
THEN rv^,•:-lude that RULt: was 

"Col vo-For-!/i tatom" witn the following 
CAVFAT: IP"^ the value of //I, when all occni'r^erioes of 
^3 are Pl-PLACE-d :v , is NOT CgCAL lo the 
corrv^sponl int^ value of ^^2, then a stuH-r^t error 
3!]. -.-^'-t the M.]h-ri:les: 
' ] I " tr an .- rose all ter-j s involving // 1 to 1 ft " ; 
,?) "lrar;/-pos'"^ all ter^:^ not involvin.- zo the 
r i h r " ; 

(?) "add like terns on each side". 



: lip 



P:;ut.. i nes should n • oo be c onst r oc t - d wh 1 ;^ h prov id o a 

o t, on' -i i d e so r ] p' i -) n of the : f f er enc es be twee n t. w ; 

'.,o-SL 'ir^s . These ni-^hi. be based an sucb featu^'crs is: 

r o r t ^ n u m b ' s : r a ^"i r i - t. ii o s e s ; differ e n t: n \: m b ►'^ ! ' 



*?Wit-b all doe arolori^s for th^ o-yptic o r te rn-rria tc hr- n 



t (MMiK'.; <\\ I'f^M'onl riumt)or\s of Lafiibda (bound) variables; 
[) r e s o n c? o oV " = " r> i |/ n ; o t o . 

# 

Anal yt i r. ^prrof^ch : 

Thc' ^)t nor . ^>[)ro:ich (i.o. analytic) that we have been 
pursuing con(^erns a careful analysis of the knowledge to be 
ornbodded in t.he tutoring system. It also includes a 
speci f lontiop of the kinds of logical tasks the . system will 
havo to perl'OMii in analyzing and critiquing a student's 
.solution. These 1 n vest 1 pa t ion s have been empirically driven 
in the .sense that we have run several exper^'ments and 
perfor^ned (by hand) detailed protocol analyses (akin to the 
methodology ^'^'f Newell Simon) of the solution paths and 
er'rors generated by students. 



In order to provide an intuitive grasp of the kir.ds of 
knowledge and reaooning involved in con.:tr uc ting a 
^•tructural model of a student. 's befiavior i\ solving an 
algebra problem, we have annotated an actual subject's 
solution path to one of our experimental problems. This 
indicates the kind of reasoninr required to construct such 
models. We stress, again, that it is relatively 
s traigh' t orwar I to have a C' ?puter system p^enerate a 
reason::Dle solution to this probl . The caallenge is to 
adeouatelv form-ilize the reasoning strategies and knowledge 
that ■ utors use in understanding what a student is 
(proba.jly) doi^.^.^ so that simila'" sti^ategies can be employed 
by a computer program to enablp it to. figure out the 
studentV*^ reasoning processes and p^int'^^ o confusion. 

Tf.^:^ al^^-'braic problem this subject has been asked to 
solv;- is to reduce an alfT'^-braic expression .Line LO below) 
to a minimal ror^r; (i.e. a single reduced fraction). l,--t 

us proceed to generate a hypoth tical explanation o^ what a 
good tutor mif?:ht be thinking wh i . e grading or unde '-^t '-ding 
t h i. s s u b j ' ^ ■ • s so] u t i o n s . 

[LO] { A L ( :KA ' ) t AA - } + 



ERIC 



The analysis of LO through L2 is straightforward, but 
the transition to L3 is puzzling. The inversion operation, 
which brings the into the numerator, suggests a 

moderate degree of sophistication. But the (very common) 



illegal cancellation 
There are some clues 
narrowing the range of 
olicit this orror. 
(2A-.3H) f ?A/3B) , which 
difference appears to 



indicates considerable confusion, 
which help to refine the description, 
contexts which would be likely lo 
Note ^ he subjecl: did not perform, 
is a very sim^1?^r mistake. The 
focus on whether the sum of torms 



oocurr, in the numerator or denominator 



U A 



[L5] 



In going from to L5, the exponent of "A" is lost. 
This could be attributed to either a copy error, followed by 
a l:.!;al factoring, ^ - a factoring error. A flag should be 
set \o watc'-: for further occurrences of the latter, but 
there is not enough information here to certain,. 



[L6] 



J. A 



(. A 



L6 is truly remarkable. The first expression, (to the 
left of the first was p ^rived at by "multiplying 

through'^ by 2. This is an in> ance of the more general 

iiffirulty, o^^erating on expr oions as if they were 
egun tions. One ./onders whether : n,. was written first 

[3c that its presence triggerpu the error), or, whether the 
Invocation of the ( inappropridle ) rule, (since -ts correct 
use occurs in equations), triggered the sequenc- of 
si^ns on this line! The second expresrion , 2A-1 + (2A-1 ) , 
could be either a '^n: ui t i pi . 1 ng through", but an abortive 
attempt to combine into a sing j fraction, :sing an LCD-like 
r^ule but forgetti-- th^r denominator. The boxed answer, 
iiA.--2, must have see-^^: strange as the subject continues on 
tn '"^actor" it, introducin -et another (slrh) error. 



What can b-- made of L7? Clear 
and the subject- roturn.d t L5. 
.:on^.Duter to be sensitive olue: 



■ tice 



c-ed i n 
tie nee d 
' kt r ack i 



' o r r • 
for 



the 
L7 



-83. 



iii .i ri'^t. liciv<^ bt.M.Mi h\i)\Aj'n Vrom L6 by Tiny j • i a u i o 1 :'ooihm'mm» 
•} 'r':!! i onr. ; t.ho bc^xptl an.swer, roilowou oy -iiid i I i ori:i 1 
■^r"k; aiiM t.h<' i.:L)lt:il. history 1 n forma t Lor' . rept> 1 1 1. ion o :' 'i 
:m^v i on.^- 1 i . ) 



',."0 



i-"^ t.,r; corroborate bhe claim that the .'ral).cct has 

• 'r"i'>ii.s [■■■r' [tl^'MH.- with combinini^ ['ractlons! Prorabiy, he 

n i '/.o rj t,ri;. :n'nLlarity to !.,6 nnd abandoned the ori\ort 
I t r: Mit >rn [) 1 M t i. riJ"^ the lirit^. This annly:;. ; ^ is supported by 

thf- ;bic(^t-, has backtracked as far as His lack of 

(mflderuT' i n the subsequent work (L3~L8) j :.. sou^d -~ he has 
••njrp'^d t . ■] correct solution path, with no_ err nr^ ab^ve 
h i, r. L-fMrv:- : the "pr'oblem behavior p;raph". 



:c:^itivc s ' , e p forward; 1.11 Ls lep;ai, i 
•\- [':>■■■. The 'Mj^ ;*^''t was apfvir-^itly 'iw^ro j\' thr-' 
r. c i, ^ ' if ^ ■ -^'^a t ^-^ I y? ) cho^' - - - a :i i f f-'^ r^n t 

Mj ifc^'^nd* '^:c: 3u -'^5^ t, - th.r. ,.s st r ater 1 ^ 
•rf :n':>re f ; j nd am v t) t n i than the rn'^tter of illec-a: 
n r» k t r ac k i rif^ , he indicates at len ■ t- a s-'';S''^ 
■ i b u * t \ 1 1 e £■ ".i \ m ''^ s . 



; r, i: r n^^ 1. : V e . 'i ' c :.c''and or ] n r 
: • fa to I i nr o f t he ; >'nom i n 'i t ' . 
' :r- f r-miss: .n, A ;^nod tut.cr^ 
:vir'a I 1 r between f ' and L, 10 : t:;'- 
d to :n c'^ dor Stan of some hivh-'^r^ 

vv ;:b 1 c , sc 1 V i ;i , t h w ■ ^w.. o -i 



i I ; 1 i '■) n [ ' . 1 1. h » 



A • 3B > :^Ja ^e^; 



(M-- 1- : 1 II t, i on 1) f 1^ fi'" '1 I t, f > rMM! \)rn\) ] C'm , 
f.>w til'' ^liffic^ult i^.:- (Viooun t>^^ r(Mi in 
r. i)t, or 'nil::;., bo 'ih t.-) df^'no. 
ijf] pri • o i'f^ i Ml': '\ 1 - : 

; 1 I r p h (') ci y o T o x ; "rim o r t ') 1 i 1 1 , 
; :\ ) n i^r'^wi rr nurnl'-r- i rid i v 1 " u'l 1 

» > 1 V ) com p I ox i 'n p 1 I 1' i a ' 1 on p ■ l (V!1 , 
frn:-; t. oollf/.- h-^v^ ^o^n.-ii :1 mat. a 
n 1 . -J 1 p r o t. r) o n : . ; p o rii * . w o t p s t: ( ^ o u 
■ ! n \' :>rol> ] ^ 'Ti ]') o r r"" o w o d '.^r ' i d a p t e d fr<: :ri 
i ti n'-:\r^r t-'-'i r:r": i 1 i t. at." c^.^fn[iar^ i .^ons ) , 

'):■;■; ;i '1 ] on;j 1 f, O TChr'rS 00 I ' ' ' '"^ . A 



•") r 



1 r 



a : T p 1 i f 
i X 1 n a ' o 1 V 
a [|^' r s 



) t i ■ : 



' r r r- aii i roa h^'- n 
a 1 1 ra : 

:';r i s i rif-^ 



a k 



? 0 1 1 ^a: " , a r o t a r i " a , and 
• 1 ol 1 Ty \ : . . a; ■ dU'^ ' 



Qualitative Impressions 



Cne overriding impression we got from this experiment 
was a sense that each person approached solving the problem 
as if by ^*rote", with most people failing to see explicit 
decision points in the solution space. Many people found 
this task very difficult. For example, out of ten college 
students presented with the simplification task, three Just 
"threw up their hands" and said they couldn't do it at all. 
The other seven tried, but none succeeded in correctly 
solving the problem. Nevertheless, the problem is well 
within the bounds of high school algebra in difficulty, 
especially for students who do not instantly panic at the 
sight of such expressions, realizing, instead, that the 
problem can be decomposed into a sequence of rather simple 
steps. 

It was also interesting to discover the wide range of 
answers to the problem (indicating the futility of attempts 
to specify ahead of time a comprehensive set of incorrect 
answers). Also the few people that solved the problem 
correctly (approximately 25%) generated an incredible 
variety of correct solution paths . The high error rate (or 
the low mean free time to an error in a solution) also 
indicated the ineffective use of student time, when the only 
feedback concerns the validity of the answer , as opposed to 
commentary on the intermediate steps which were performed. 

Figure ^.2 below shows the range of answers generated 
for the problem and Figure ^.3 shows a typical solution path 
leading to the correct answer. Figure ^.^ shows an 
annotated solution to the problem which points out some of 
the decisions that could lead to solving this problem in an 
"optirr.al" way. 



[insert Figures ^.2, i| . 3 , ^-^1 



Final exam data: 



The analysis of the final exam data from two college 
level remedial math courses involved using several 
strategies. For both groups of exams, every problem on 
every exam was re-graded (they had already been graded by 
the teacher) in order to understand some of the heuristics 
used (and errors made) by graders in real world situations. 
The solutions for three randomly chosen students were 
analyzed in detail for every problem, carefully describing 
and accounting for each error. Furthermore, a single 
problem was selected and analyzed across all students, in 
order to estimate the range and variety of rules and errors 
which can occur in even a short exercise. 
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FIGURE 4.2 



Some Typical "Answers" 



2A~3B 



2-3 (£) 



1 1 

A ^ I 



3B-fl 
4A-6B 



12AB 



J; 1^ 

18B2 2 



3B-fl 
4A-6B 



3B 1. 
2(2A-3B) *^ 2 



A 



4 (A-A)-9 (B.B) 2 



1 1 I:^- 

2 • 9B 



2A+3B 
4A-6B 



A 1_ 
2A (1-3B) 2 
3B 



2A-6B 



A 



2 (A-3B) 



-1 



1 / 1 

2 



3B_ 
"2A 



A 1. 
2A (A-3B) 2 
3B 



1-f (2A-3B) 
2 (2A-33) 



1)8 
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Plfr.ure 'I . 3 



This is a typed replica of a subject's solution path 



Reduce 



|a -i- [(2A-3B)(2Ai-3B)]} + I 



|f(2A) -.|f(3B) 



'IA^-SAB 



3B 



X 3B 

x^r(J4A-6B) 



3B 1 
'IA-6B 2 



3B , 1 

2(2A-3B) ?. 



3B 2A-3B 
2(2A-3B) 2(2A-3B) 



2A 

2(2A-3B) 



A 

2A-3B 



9 9 
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We firnt. transfor'm the problem int ) a two-dinien;; J onal rormat. 



A ^ 1 



(2A-3B)(||) ^ 



2 A \ 

At this point we could either distribute the (^g) across the 
'^PA-JB) term or cancel the A^s. Cancellation is cnoson. 



iyA-3B)2_ 2 
3B 

2 

Again we could distribute the 2 or but we decide not to since 
the matches the | on the other fraction. Hence to accentuate 
this t7;lcbal match we move the 3B up. 



3B 1 
(2A-3B)2 2 

v;e are now in a position to combine uhese partial fractions into 
one fraction. Taking advantage of the 2 in the denominator of 
eacli fraction we can combine them as: 



3D+12A^3H 
(2A-3B12 



2A 

which leads to ( 2A-3B ) 2 



arid then to 



(,2A-3B) 



erIc 



oorno ^.^lobal obr.orv a t i o n vS about this data are in order: 

(A) People are lucr ed ibl y bad at algebraic 
manipulation. In some sense, we were aware of this 
before we started — but our estimate of ho w bad was an 
order of magnitude too naive. An example may help to 
illustrate: i 

On their final exams in the college rr.medial 
alp!:ebra coijrse, sixteen students were given tne 
following problem: 

(-3X) ^ (2X) 

Not one stude nt in the course obtained the correct 
answer, '^72X^". The most common answers (four 
students each) were "72X", and "-65X5" Other 
variations were: "-36X" , "1 TX^" , "-216x2" , "-72", 
"-72X2", "_72X", "-108X2", and "-16x2+6x3". Results 
for this problem reached by students in other courses, 
from other schools, were similar. 

(B) A close look at examinations which had already been 
graded by algebra teachers has indicated fundamental 
inadequacies of traditional grading techniques. By this is 
meant that the ranges of scores which would typically be 
assigned on the basis of right, wrong, and partial credit 
answers are only crudely correlated with any reasonable 
metric one might wish to apply to the knoTjl edpc e s tr uc t ures 
in question. Suppose we describe in detail the knowledge 
which we desire for the student to acquire and use. This 
might be formalized as a set of procedures, operators and 
heuristics. Perhaps the student has accurately learned all 
but one of, say, fifty procedures. But in applying this one 
"buggy" procedure he consistently makes some simple mistake. 
If there are only a few problems which require the buggy 
procedure, the score will be high. However, if the faulty 
procedure applies at a "low level" or -- is called by higher 
level procedures - (which by contrast implies that the 
student's high level knowledge may be quite sound), the 
procedure will be needec in many problems, and the resulting 
score will be low. Similar themes are elaborated upon by 
the following examples: 

(1) A common test grading heuristic (necessary if 
there is a poor faculty/student ratio) is to exam in e 
the student's work in detail only when the final 
answer is incorrect (usually to determine wh ether any 
partial credit should be assigned). Those problems 
for which the correct an swer has been obtained are 
only cursorily exc. mined (usually to ensure that no 
cheating has occurred). This claim is based on 
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oat'ol'ul r'('^2:rad ing of tests proviou.sly graded by 
algebra teachers for the purpose of determining final, 
credit for a course. Wo can provide examples of 
students who were passed on the basis of this 
heuristic, whose alf^ebralc skill appears to be 
inferior to that of some students who did not. (Other 
trrading techniques also led to the opposite 
situation.) Two trivial instances should helf) to 
illustrate how this can occur. According to a rough 
statistical analysis, one student alwnys guessed when 
assigning the sign after an algebraic transformation. 
Since the correct answers reflected even parity about 
-ns often as odd parity, this strategy was remarkably 
successful on short problems with few intermediate 
results. This student did almost as well as several 
others who sy- temat i cal 1 y appl ied "almost right" 
procedures. A second, very common confusion which, in 
more difficult situations can lead quickly to compound 
errors, involves the relative precedence of the 
various arithmetic operations. Students who 
oo nsistent LV misapply use of precedence ^ and 
consi st ent ly make inappropriate use of (or fail to 
use) parentheses frequently end up with correct 
answers, even though many of their intermediate steps 
are incorrect! 

(2) Very often, multiple errors will occur in the 
solution of a single problem. This reflects a number 
of weak areas in the student's knowledge. For 
examplp, the work may indicate use of the following 
faultyVules: "X 0 => X", "(XY.-Z) -:- Y => , 

»tA/B + C/D => (A + C)/(B + D)", and so on. But the exam 
score may be only slightly worse than that of a 
student who only manifests the last difficulty, since 
the first rules may only be invoked in problems where 
the last is also needed. 

Additional points that have emerged from our 
empirical analysis include: 

Some of the exams were "uniform graded" by algebra 
teachers. This means that the same problem was graded by a 
iriven teacher across all of the exams, but different 
teachers would have graded different problems by the same 
student. This provided an excellent opportunity to assess 
the effectiveness of analysis which can be achieved by 
humans in the absenoo of a detailed student model (on the 
assumotion that constructing an adequate model would require 
analysis of more than a single problem). In fact, there 
were^ many instances where a more global view of the 
student's strengths and weaknesses would have facilitated, 
no^ only the assignment of credit for the problem, but the 
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utility of romedinl comments which oould have (or oin^ht to 
have) been made. Whereas many students would ciotually be 
required to repeat an entire course because of^ their poor 
scores, a global view of their work on the examination 
indicated that a few sessionr, of precisely planned 
remediation mip:ht fiave sufficed. 

It would be fruitful in teaching algebra to maintain a 
distinction between two separate pha se s of trans form in-?: an 
expression. The first phase consists of procedures for 
parsing the two-dimensional notation into an unambiguous 
internal representation (i.e. perceiving the structure of 
the expression). The second phase consists of the 
application of "algebraic" (as opposed to "perceptual") 
procedures to the internal representation. Many student 
errorr which we have seen can be more succinctly 
characterized as the result of mispar sin^ or misperceiv inf^ 
the external notation, than as incorrect applications of 
algebraic procedures per se . Consider the following very 
simple example. The student is asked to evaluate "XY", when 
X=-3, and Y=-5. The student writes, "-8". This can be most 
simply understood (especially in the context of many similar 
errors) by assuming that the student performed (perhaps only 
in his head): "-3-5 => -8", not having understood that the 
notational distinction between concatenated literals and 
concatenated signed numbers constrains one to employ 
additional oarentheses during substitution. Note that 
little is ever explicitly taught about how to proceed in 
parsing (or, for that matter, carefully and unambiguously 
svnthesizin/^ ) algebraic notation ! As a corollary, when 
understanding student moves, the system's internal 
representation must not lose the (geometric) information 
needed to propose that such an incorrect parsing may in fact 
have occurred. Another example of an error that most likely 
relates to a perceptual - le a rning problem rather than as a 
difficulty with the mathe m atica l pro .edures per se is tha^ 
manifested by a subject's work shown in Figure ^! . 5 . 

[ insert Figure ^ . 5 ] 

It must be em pha si zed that the tutoring system must 
embody sufficient algebrai c expertise to deal with fairly 
complex, unanticipated situations. It is quite common for a 
simple error to convert an elementary problem into a 
relatively difficult one. For example, one student, when 
con f rente d wi th : 

3 4 X 

Solve for X: ^ ^ = ^ 



wrote: 3 2x+2 




x+1 x+1 
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(A) Student's work: (i^9F) 



(B) Explanation as a doscription/perceptual-learning problem: 



2/x + 3/ y 4-2 / xy 

I V 



(This is conventional parse) ^ 

(This is student's parse.) 
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in which a single sign error has converted an e 1. emon ta ry 
linear equation into a quadratic, which could easily overtax 
the student's abilities. Such transformations occur 
frequently, and, moreover, provide the contexts in which 
student expectations lead to further mistakes (which tend to 
restore the situation to something "within normal 
boundaries"). The following student's calculation is a good 
xam pi e : 



Solve for x. x+9-3x=2x+l 

2x + 9 = 2x + 1 (commits a sign error) 

^x -2x 

X + 9 =1 (his subtraction is in 

correct but is explain 
^ 2^ able ~ see text) 

= 8 



In order for the system to intelligently field such events, 
its algebraic competence must be extensive enough to 
recognize the student's dilemma, and not limited, say, to 
hand coded solutions for pre-arranged problems. This is 
only one of the ways in which intelligent CAI needs to be 
"generative". In this particular case this expertise must, 
indeed, be fairly extensive, encompassing a "theory" of how 
expecta t ions — established by the input patterns for a 
procedure — can affect one's quick perceptio n of the 
problem. In this pr'oblem, there is little doubt (having 
viewed the students' previous work etc.) that he "knew" that 
2X-2X=0. That is, if he had been given the problem to 
evaluate "2X-2X" in isolation, he would have given the 
correct answer without hesitation. However, this sane 
problem in a context which sets up powerful expectations 
(cGiiCerning the pattern for solving such equations) led him 
on the one hand to conclude 2X-2X was X and on the other 
hand t ha t it was 1 . 



i 0 .> 
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A vci-y l.ifK,.- pr'oportion of the; orrorr, which wo havn 
encountered can be aocounteci for by at. most several, dozen 
error sehemas; in other words, a relatively tight error 
taxonomy can be employed. Our cateRor ization of common 
errors seems to be almost universal. Our data stronp;ly 
supKest that the occurrence of the various error types is 
independent of the student's particular oackground (such as 
which high school was attended), as well as of the 
examination score, within a fairly wide range. Of course, 
very poor students turned in virtually blank exams, which 
did not manifest these errors at all. Likewise, those 
students whose scores were almost perfect (there were one or 
two) made on""" those errors which could be described as 
"simply carciess". But for the overv;helming majority of 
students in the middle range, the following sorts of errors 
were typical: 

(a) [anything * zero => anything]. e.g., 

5(-5)2 (-4) - (-5) (-4) (0) = ? 

-500 - 20 — > -520 

( b) [a + be => (a + b)c] . e.g., 

,2A, 4A^ - 6AB 
2A - 3B(f|) --> 3B 

(c) [anything -:- zero => either zero or anything], 
e.g., 

{7(-5))^ 1225 ^225 

r a c _v, a c 1 

(d) t b "■" d b ■ d ^' There are several 
subvarieties of this form, but this is typical: 

I 3 5 15.35 

(e) Illegal cancellations of the form, 

r UI^'CL => x+z ]. 

y+w 

of which a typical instance was: 

oyi 5 6 8 ov2,4 
-24x y z -ox y 

,328_328 -3 
3x y z -9x y z 

(f) incorrect versions of fractional addition, 
taking forms such as: 
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^ b d ^ b+d ^' 

Til i i '^rie^ of t.h^^ \]\oi'\l oommot) d i f'f; ''^li It. i , -imU 

t, lirt'M ;ivr m:\\\y v;i r i 1 1 i onr. ; a p[).u'(*n 1 y t.hry rM'Tl'^'t, 

V o r'rn o n t , h 0 b n I r. o f ' ; ) r 1 rn a r' i 1 y : ; y n t ; w ' t i f • 
(.: o n i 6 e r a 1 1 o n s , S t u d n t x i\ mp] o: 

2 - -3.. ._» -_i 

x-1 x4^:^ 1 

(r) Partial d i str' 1 but i onr, r>uch a53: [A(B +/- C) r> 
An +/- C], which include;-; the npocial case: 

-(3x-w) -->> -3x-w 

(^1) Confusion of arithmetic operators. Example: 

Note that several of these (such as b, e, and h) can be more 
easily explained as attempts to provide interpretations for 
notation with which the student is unfamiliar (or has 
forgotten). This list ic illustrative, but by no means 
exhaustive. A more complete list should provide the 
doma in- spec i f ic basis for the system's remedial stratef^y. 



f'x pe r i m e n t a 1 Exam 

The results of analyzing the final exams of the two 
remedial math courses raised some serious doubts as to 
whether the kinds and frequency of errors we discovered were 
due, in p^rt, to the lack of any semantic basi5 for the 
mathematical orocedures for *^how to do it" type naterial) 
used in t h e 5^ f -courses. 

Part, ly to explore this possibility, we designed an 
experimental t'^st comprised of problems from each of the two 
exa'ir^. 'jsed in the special remedial classes. Tiis test was 
then t^iven to a group of college students ( f r :)m a different 
CO 1 1 e^e ) who came from a i verse background s and high schools 
in which a variety of teaching styles and ".ext books had 
''ri used. 

The data of this experiment were subjected to the same 
. ; r,,i of analysis as was used in analyzing the remedial math 
' xams. From this analysis w^ discovered that, in fact, the 
.:inds and to a considerable ' .vtent, the relative frequenoi'^r 
r the nrrors. encountered on t'.is experimental exam were th'^ 
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■ 1 1 1 f 1 ^ • M ■ ■ n » • • iir pr i :■. i fir 1 V ! f ; >1 ^ ' p' • n ■ i t «f 1 1 - [' t h^- t • h i nr 
: ^ V •u-l "I \ \ ^ \' 1 1 . * 

♦ h i 'hi -r' . ' 

,\ I'f'i' 1 i m i M V T:i"^'r"V 'I" Mint.', P>jr;i, Uhl H< 'nn -i i j t, i n n >r' 
M'^'.lijf I . i 1 1 in A I )■', ♦ ' ^ ' ^' ■ ^ 

u-p,>Mj t n^' {t-'-v X ■ r' i T rn t , in cmwi j i]n^' t i .■)n wit,fi a 
: 1 i Ml i n.in V .-.tuiv 'M' 'J pt- '(MMji.rn 1 knowl^'d^v* ba:w' Cor 
,l;,,,pp;,|. ':i>:i{|-, hrii^-vo t hn t a thoory oV hint.:'. and 
-orrN'-l i It i iMi will [.:'■ >t)ai' 1 y rir-l t r) bp ^'lor.rly t. i ^mI lu a ;>tudy 
'au.'-.alitv ind (M]r'[)*)V.'' { 'Ma- Mm . 1 o^'; y ) . Mwiy of our^ 
i r) i .'lit. n irita t h<"a> i:;:;uf;- in*- a 1 ;u) (I'T'iVv: from rcrMMit. 
rMV';M ,r'.-h >Mi nn Ira'.-.t and i nr. , P' 1 arm i nr , and <jt^bur,/^ 1 riK 
itir^T. . ^-'ravn 1. h i lat.M^r' wor'k , it- af^a^at^r. r.hat. many 
•innta; )r-- ''-arM < \.n a, t r' ua taj r a 1 f*.' :i tajr r.;; prooodural 
r'oprvvaM]t,a ^ i -nr : [) r'oc o n ci i t iari:^, iJllClQ'lL': c owm^-^'^ - r V 

; " p . V.M tM c 'I t. i ' Ml : ; " ) ,.Lll,!::^in t, \ r v. } ( " i n cj u o t. i v ^ a s o r ' 
i nt'-^r-rf^or'-'-l wi_Mlili ^ laaMnrrita;), '^xAM riM ^'•orr 

(pur^v;>:a'r wliaC hirlior l.ovol ^oal;;; arr* beini^ iunicv 
'iunal. ( i.^: arm i faO oornrnontary (how t.he pr'ocodure achieve 
""^h jo-fi V'*:'"i . Lik'a;i.'a^, romp:! i a i on is olosr^ly linked t.o 
i^r ^c-^-Aur -\\ "C.■^v£al^l" wliioii warn at^oat t-ypical bugn, 
■•nv i rr n-a^r. * a ;n"""wh:''h the nr-requ i f^i ta-::^ r?i:iy fail t-.o ho 
j ..p; '-^r. !n flat, ■^ir ataJdy of the teaohinp; '^nd 

n - r ] I \ '^^] au-ri rmr^'" -d ni'a 1 knowledge or'omi^pa ^o lead to 
>ajr:aa'^pr.-/l. inrirhts ir.va auoh issuers as how "bij-^r^' aru^-^e 
fp'.y .^r. :.'r^..r- i vpI V -'Xt.eridmr i>rocedure3 handl*^ .i wider 

I opv i r. aim^^nt ** A r i t. hm ^-^ t i c ■ Derations, 
~, r.' .«v-,-f.: , ir- firr,t learn*-i wlr,h r'-.spect to the r'ttural 
Mi--<jr> ' i n.r n rnr^her:^, ^- hen the {s.^ned) integers, then 
1 vr.' -1 i '1 1 )0 w^-*ll th*' rmc r r 03:^' i o n throuf^h rational 

, i r.->a| 'iard'^era, etc.). Many of the oommon 
" M . ,r>r.:•;^p;: r- V bo ] ' a'' 1 V ^ ' d frofii thla aequence of 



a : "i : 



■ r... r.;: ] ; r, - - ■ o : < ? 1 vt 1 c and 'v/ri M.^' 1 1 study of 
:a -f r:nowIv'^^^e , we aow ir^'^ :n a position to 
■1 r)ra''ti':^al I'^AI aystp'^ for nr'oviiinp: hip;hly 
t^-^'l 1 i .^ent tut>:)r : n.^ for al 42:ebrai ski lis. The 
-^oarlv indioat^^ M.^^ need for the tutorinp- 
T "j : a 'J " ao ph i St, i a a tp i ;;r'o h 1 p'^.- so 1 v in e; component 
*r-]yr. .-^ V r'^ I i '-^ i ^ all ' h^ irnr^li^'it steps that a 



1 r'l 



M-.- 'n':--'rit: n::' 1 U'i o th:'.* -ir; inabili*:;/ to 



';**)This idea oriqmated, in part, from Ira Goldstein's compcllinq 
theory -^x Rational P.nus as devob.)PPd in his doctoral thesis. 



student performs in ,his head while solving a problem, and b) 
to characterize and generalize the underlying cause of an 
encountered error. These preliminary studies indicate that 
a problem- solving component with such capabilities is well 
within our reach given the kinds of errors and error types 
we have encountered and classified. Our next step is 
therefore to build this component and to couple it to our 
various tutoring strategies. 
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INSTRUCTIONS FOR EXPERIMENTAL ALGEBRA QUIZ 

This quiz is being given to you as part of an experiment to 
understand how to teach algebra more effectively. Your "score" will 
not affect your average in any courses you may happen to be taking. 
SIMPLY DO THE BEST YOU CAN. No particular weighting will be assigned 
to the problems, although they range (in order) from easy to fairly 
difficult. You will have one hour to work on the exam. You are not 
expected to be able to complete it in this time. Try not to spend a 
great deal of time on any one problem - if it seems too hard, go on 
to the next. There are four pages, not counting these instructions. 
Although not overyone will have the exact same version of the quiz, 
they are all about equal in length and difficulty. Do not attempt to 
work the last two problems unless you have considerable time left 
after completing the others, and checking your work. Please curn in 
all of your work! Thank you for participating! 
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1. (-3X)^(2X)^ = ? 



(-5) (-3) (0) = ? 



4 4 
3. (-2) - 2 = ? 



X 

0 



TRUE OR FALSE: When X=2 , Y=l , and Z=~3, 
X(Y-Z) (Y+Z) = -16 ? 



2 

TRUE OR FALSE: (3+4) = 9+16 ? 



Solve for X: 



0 3X + 24 = 18 - 3X 



8. Multiply: 

@ XY(-2X+3) 
(b) (X-1)(X+5) 
(C) (X-3)^ 



1 11 
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9, Factor compJctely: 

© + 5X + 4 

(b) 3x" 4X + 1 



10. Reduce to lowest terms: 



2 

XY 2 
X'^YZ 



4X-4 

2 

X -X 



..568 
-24X Y Z 



3 2 8 3 2 8 
3X Y Z -9X Y Z 



11. Combine: 

2X'fW _ 3X-W 
3Y 3y' 



® 



^ X-l X+2 



12. Evaluate, when X=-5, Y=-4, W=7, and 
@ 5X^Y - XYZ 

5XYZ 



13. Perform indicated operations and simplify 



15_ . 3^ 
7^4 



1 3 



5 
6 

1 12 
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Solve for >: j 



® ^ 



+ 2 



X+1 



© 



3X4-1 _ 5 
2X ' 2 



(OPTIONAL) Reduce: 

fA 7 C(2A-3B) (2A73B)3) + 1/2 

(OPTIONAL) Solve for X and Y. 

x-y = 2 

2 

y -2X = 4 
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CHAPTER 5 

TOWARD A MORE FLEXIBLE INPUT MEDIUM 



Existing constraints on how a student can communicate 
his thoughts to a machine have unwittingly shaped our views 
on the range of tutorial interactions and strategies that 
are possible in CAI systems. The effects of these 
limitations are abundantly clear in the various mathematical 
teaching systems in use today. Simply stated, algebra is 
not naturally done on a typewriter! The kinds of algebraic 
expressions that a student manipulates cannot be 
conveniently represented (or even thought about) in a 
one-dimensional medium. | Mat hemat ical expressions are 
inherently two-dimensional: exponentiation is represented 
by superscripts, fractions are represented by vertical 
arrangements of numbers and the fraction bar, etc. In 
addition many of the manipulations in algebra are described 
in terms of a two-dimensional layout of expressions. 
Consider cancellations. Without a t wo-diinensional 

representation the rule for cancelling the same factors 
above and below the fraction line would h^ve to be expressed 
more awkwardly and then relearned to be applied on 
expressions written in standard two-dimensional language. 

To avoid introducing these artificialities we have been 
designing a two-dimensional tablet based input system for 
the algebra tutoring systen. It is our intent that with 
this system the student working on an algebra oroblem can 
use this tablet input mechanism just as he would se scratch 
paper for working out his intermediate results. This would 
allow a student to work in a natural way (i.e. just as he 
would without the computer) This increased view of the 
student^s work improves the bandwidth of information being 
passed to the student modelling; system. It actually allovro 
the computer to look over his shoulder and watch what he is 
doing in order to provide hints and develop a better 
understanding of the processes that the student employs. 
For this to work, however, the recognition system must be 
robust and natural to use or it will interfere with the 
thought process of the student. 

The task of recognizing handwritten algebra expressions 
can be thought of in two parts: 1) the recognition of the 
individual characters, symbols, numbers, and digits, and 2) 
the putting together of these symbols into larger 
expressions, that is, oarsing the string of characters into 
mathematical expressions. As mentioned above, this parsing 
must take into consideration two dimensional information, 
since many of the algebraic operations are expressed as 
two-dimensional relationships. 
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Before going on, a word of caution may be in order. 
The concept, and some of the techniques, for using a tablet 
to do limited character recognition of mathematical 
expressions is not new. Indeed one of our prototype systems 
(to be described later) draws heavily upon some of these 
existing techniques. However, nearly all existing systems 
have suffered from two constraints: first, they were 
designed during a period in which dedicated computers of the 
power of LSI-ll (PDP-ll costing around $700 ) were inerely 
pipe-dreams and hence the designers of these systems were 
forced to make many compromises solely because of the lack 
of computational resources. Secondly, and much more 
importantly, none of these systems had any effective way of 
using higher-order knowledge such as semantics and 
pragmatics of the given domain to help resolve ambiguities 
that inevitably appear in the input. In fact, simply 
segmenting a stream of character strokes into meaningful 
characters often requires knowledge about the likely intent 
or meaning of the expressions, as does coping with any 
sloppiness in the way these characters are written. The net 
result has been tablet based input systems that are awkward 
to use and require a substantial amount of human 
adaptability . 

Another reason for our exploring how to design more 
flexible and context-sensitive input mechanisms, concerns 
our belief that a knowledge-based CAI system must be able to 
use its knowledge to perform various tutorial tasks as well 
as regurgitate factual information. It seemed to us that an 
elegant and natural spin-off from having systems that can 
engage in tutorial reasoning would be that these same skills 
could be focussed inward to he! provide a dramatically more 
flexible and "forgiving" input medium. Indeed many of the 
same techniques for isolating, describing and even 
correcting student errors are precisely what is needed for 
deciphering ambiguous scribbles. 

Experimental Input System: 

During this contract we have devoted approximately one 
man month to constructing a prototype system (that is 
currently demonstrable) and the design and partial 
implementation of a second more ambitious system. Both of 
these systems use the IMLAC, a Computek tablet which is 
interfaced to the IMLAC and a TENEX computer. The IMLAC 
does the local and real-time processing of the character 
strokes as they are drawn on the tablet. The IMLAC then 
passes the parameterized stroke information off to the TENEX 
where nearly all the computation is performed. Our 
prototype systems are programmed mostly in LISP which means 
we have sacrificed real-time performance for an extremely 
flexible and powerful programming environment to develop and 



-111- 
118 



-'X pi?f*imon t. with our idonr,. It. is our intent that af*t(?r' wc 
iiavo refini?'] our^ algori th-nr, mkI control strategies vjl^ sh:ill 
then worry about im pi fMruMi t. L n^"^ them on a system .lofrif-^tl-M ru'; 
like LSI-1 1 . 

Our first prototype system was written by Daryle Lewis. 
This system uses a character recognizer based on the Ledeen 
algorithm which is described in Pr i nci ples of Interactive 
Compute r Graphics <Newman & Sproull 73>. This character 
recognizer is also trainable. During the learning session 
the recognizer extracts simple features from each stroke and 
stores the character in a decision table under these 
features. Very briefly, during recognition it determines 
the set of characters that the input character might have 
boon by comparing the features of the input character with 
the features in the decision table. 

This initial system also used a top-down syntax-driven 
parsor described in Andersorfs <Anderson 68> thesis as a 
technique to use some higher-order knowledge in interpreting 
the input., In the appendix of this thesis, Anderson 
describes an efficient parser for mathematical expressions 
which is a specialization of his more general parser taking 
advantage of certain features of expressions, namely that 
the principal operator of an expression is always the 
left-most character. It also uses the stronger assumption 
that arithmetic expressions are almo st linear , and that they 
can be linearized. In fact, the scheme begins by 
linearizing the expression. It does this by first sorting 
the characters by their spatial location, the , assembling 
string/s with the notation for superscript and subscript. 
Arithmetic expressions can be described as linear with the 
exception of fractions and as previously mentioned, 
subscripts and superscripts. Then a linearized expression 
is parsed in a conventional top-down, left to right way with 
backtracking. Lewis's prototype system currently recognizes 
arithmetic operators, integral signs, summation signs, 
fraction bars, numbers and variables. • Example of 
expressions it's capable of parsing are shown in Figure 5.1. 

This system demonstrates that a very simple system with 
a minimal understanding of two-dimensional expressions can 
do reasonably well in a highly constrained environment. 
However the system has definite flaws and probably cannot be 
extended to adequately fit the needs of our algebra system. 
Some of its problems are as follows: Anderson's parser 
never really had noisy real-world data. Instead he gave it 
letter-perfect data: two dimensional coordinates of 
characters, perfectly prerecognized characters, and 
correctly assembled integers. Unfortunately, probably no 
character recoi^nizer can be good enough to present student 
generated data in this clean a form. Since the string 
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p.irs^M^ is nondeterminipMc , It could be ext^nd^d to c.ov^^r 
t ho uncertainties present in the character recognition t)ut. 
t hP linearizer is inherently deterministic and is wry 
sensitive to poor noisy data and therefore limits the 
robustness, flexibility and strength of this kind of 
recognition system. The character recognizer of Ledeen has 
its limitations too. For exaii.ple it does not address the 
.^>t^p;men ta t ion problem, that is, partitioning the stream of 
strokes into groups of strokes making up each character. To 
oircumvent this problem our initial system required the 
.U.udcnt to pause a half a second between characters in order 
In unambiguously and correctly segment the strokes into the 
oharaoters. This technique works but represents an 
unnntural and disturbing constraint for a student using the 

Bf;oause of the psychological ramifications of these 
constraints, the system was extended to the point where it 
collects a stream of strokes wi th out explicit indication of 
the inter character segmentation and then attempts to fit 
characters over the stream. Spatial information is used in 
this segmentation. The overlap relationship between 
adjacent strokes is used to strongly suggest that this pair 
of^ strokes belong to the same character. Likewise a longer 
sequence of strokes that makes up a character is preferred, 
and a number of other heuristics help the segmentation, 
Althoup:h tnese changes improved its likelihood of success, 
we decided that a fundamentally different approach should be 
o X pi ored . 



■>iow Views of Parsing 



One view of parsing is that of translating strokes or 

other input into a tree or other meaningful structure. In 

this view, a grammar describes the bridge between two 

representations. The meaning of an input travels over the 

bridge into new, structured representations. Parsing is 
(grouping and rearranging information, 

A more recent and more powerful view is that parsing is 
not rearranging meaning but rather finding meaning, and Just 
as importantly, finding intention in a communication. 
Consequently, an input may not be a correct realization of 
the intended meanine^, but may be full of mistakes, 
sloppiness and other noise. To translate the input without 
any consistency or plausibil checks will further obscure 
the meaning by introducing ore erroneous structure. It 
is only by knowing what the c unicator could want to say 
and what could be said that ^ utterance can be understood. 
A parser and a grammar must be much more a filter of what 
inputs make sen.se. What the input seems to mean mny be 
nonsensical or irrelevant to the situation in which it Is 



said. It is only by understanding the situation that the 
right interpretation can be found. The importance of the 
situation of an input has only recently come to be 
recognized. Communication always has a context and can only 
be understood within it. Linguists have become painfully 
aware of the weakness of syntax without semantics. What we 
hope will distinguish our cablet understander from previous 
ones is the constructive use of semantic knowledge as a 
powerful constraint or filter for parsing. 

There are many levels of knowledge to be applied to 
input that will potentially help the parser disambiguate. 
A^; the simplest level, algebraic expressions have a syntax. 
vor example, the knowledge that parenthesis come in pairs 
can help distinguish a "C" from a left parenthesis which can 
be realized by the same input stroke. Only context 
identifies which was meant. N'^^ural hand printing is not 
procise enough to distinguish between these two characters 
or between many other pairs of characters. For this reason 
many previous tablet systems have required the user to learn 
new printing conver.ions such as slashing zeros. These 
conventions shou] ' not be necessary where the meaning is 
"obvious" (to the cudent anyway). At another level of 
knowledge, the .-tem knows which algebraic manipulations 
are being taught lerefore, by knowing what exercises are 
being performed, r student's input can be related to them 
by the operations cind manipulations being taught. The 
system might even know of faulty manipulations that students 
commonly use. For example if the distributive operation 
were applied to (A+B)X and the parser found AX+13X it would 
be able to see it as AX+BX. On yet another level the system 
could use its model of the student to understand the input. 
By being aware of the student's vocabulary of algebraic 
symbols and manipulations some interpretations of input 
become less likely. 31 might look like 3!y but if the 
student doesn't know about "!", the factorial sign, then the 
correct interpretation is easy to make. 

So we have seen that there are many levels of knowledge 
that can be applied to parsing. What makes this system a 
challenge to design is that each of these sources of 
knowledge is imperfect and will introduce errors. 
Collectively they must correct their errors and arrive at 
the best interpretation of imprecise input. 

Chart Parser for Parsing Subexpressions 

Using a straightforward top-down parser for rejecting 
anomalous character identification is subject to serious 
combinatorial problems because a purely backtracking parser 
throws away the good information gathered under an 
unsuccessful hypothesis. To overcome these ind ether 



problems, another approach is being tried by Steve Purcell. 
The second system uses the same character recognizer but 
will employ a substantially different parsing algorithm. 
The parser will be a chart parser like those developed by 
Martin Kay and Ronald Kaplan. The chart is a data structure 
to hold the fragments of successfully parsed expressions. 
All the phrases that are ever noticed are remembered and are 
available for building larger phrases. The chart holds the 
entire utterance or expression. This allows parsing to 
proceed from whatever islands of certainty there are. The 
parser is not constrained to proceeding in chronological or 
left to right fashion! The chart would correspond to the 
notion in Woods' ATN grammars of the well-formed substring 
table. This is a table of subexpressions which are saved 
when the parser makes a mistake and backs up. The 
weli-formed su' '^ring is a successful parse of a group of 
words. It may be used by another path in the grammar other 
than the one that first requested the phrase. This helps 
th. top-down parser prevent the repeated parsing of lowest 
level expressions. 

The basic concept of the chart aust be adapted to two 
dimensions. In addition, there are strong built-in 
assumptions that the parser is parsing a one-dimensional 
utterance. In one dimensional strings a phrase is 
constrained to cover or include one interval (in time) of 
input data. In two dimensions a phrase is, roughly, some 
continuous neighborhood of a simple-connected set of the 
two-dimensional plane. In one dimension the 

simple-connected set is merely an interval and can be 
described by its end points. In two dimensions there is no 
such simple description of simple connected sets because the 
sets can be very complicated. There can be very many 
different connected sets. 

This second system will not linearize the characters 
into a string for the parser. It is felt that this would 
destroy many of the two-dimensional relationships. A 
mapping of two dimensions into one is always going tobe 
fragile and unstable. Small differences of interpretation 
in two dimensions could lead to radically different one 
dimensional strings. 

The structure of a one-dimensional grammar or parser 
rises from the notion of phrases, coupled with the notion of 
concatenation. In one-dimensional strings there are two 
kinds of concatenation: symbols are concatenated to the 
left and to the right. Similarly, larger phrases are viewed 
as concatenated with their left and right neighbors so the 
constraint for forming a phrase in a string is that the 
constituents be contiguous and non-overlapping For a 
phrase to become part of a larger phrase it must combine 

-116- 

O 123 



ERIC 



either with its left or its right neighbor. In two 
dimensions there are more possibilities; a phrase could 
combine with another phrase in any direction. For our 
purposes (in algebra expressions) we consider nine different 
neighborhood concatenation relationships: above, below, 
left, right, overlapping and the ^ diagonals. The model of 
parsing will be to take the characters in their enclosing 
rectangles, the coordinates of their centers, and to compute 
the two-dimensional relationships between neighboring 
characters to propose neighbors as potential phrases. It 
will record the neighborhood relationship of the larger 
constituents and grow larger ano larger phrases. The 
constraints on this parser are similar to the string parser. 
Two subphrases of a phrase must be neighboring and they must 
be disjoint sets of strokes. The parse will be a tree with 
one root and the leaves will be all the strokes or all the 
characters of the input. 

Some phrases may cause expectations to be set up which 
will affect interpretations of neighboring phrases. For 
example, the presence of an integral sign would cause the 
character "d" to be interpreted as part of the integral 
derivative. The parser will reach decision points as it 
builds phrases. This reflects the local ambiguity that will 
not be resolved until hopefully later in the parse. The 
chart has to hold all the possibilities or the 
representation of the possibilities so far encountered. 

The chart is constructed so that decisions or 
ambiguities in one area of the expression will not recombine 
with the ambiguities of another, as long as these phrases 
are disjoint. This is in contrast with the back-up parser. 
If two independent assumptions or decisions are made and' the 
first proves false, then the second is needlessly undone and 
remade. In this way the backtracker multiplies out 
ambiguities even though they are essentially in phrases that 
do not overlap or interact. 

But even a chart parser still has the potential of 
exploding. There may be an ambiguity in a subphrase of a 
much larger phrase. This ambiguity could be preserved and 
larger phrases constructed which may be similar in every way 
except in one subphrase of a subphrase, etc. To prevent 
this kind of explosion we hope that the parser can converge 
its idea of the parse and merge alternative parses as soon 
as there is a resemblance. 

For example there is the ambiguity between the letter B 
and the number 13 which participates in a large phrase 
4(A + 13) versus 4(A + B) . The parser would build two 
independent large structures differing only in the B versus 
13. The Purcell parser as planned will merge these two 



jlrii>' I iir'o;; into the r,:\me structure where the ri'()resentat ion , 
the H or the 13, will have a decision marker 'TI it. When 
alternative phrases firise which do not merge soon, the 
parser will stop growing the phrase in that direction and 
wait for expectations to be established to Riive it better 
guidelines for choosing between ambiguous phrases. 

The algebraic context is another source of information 
that the parser must make use of. In a series of 
expressions written in the course of solving an algebra 
problem, there is much similarity from line to line as 
expressions are manipulated and rewritten. For example, 
there are many expressions that the parser should be able to 
recognize once it encounters the phrase 4(A + B) . If 
ij(A + 13) is then input, the 13 should be forced back to B 
or the original B should be reworked as a 13. 

In other words the parser will attemp^ to recognize 
expressions that it has encountered before. This is an 
example of how the semantics of algebra augment the 
syntactic grammar in the parser. To do this, equivalence 
relations on several levels will be recognized. There are 
many graphical realizations of any given number. For 
example, 4 can be written with an open or closed top. There 
are equivalences in the algebraic vocabulary. D^^vision can 
be indicated by a minus sign with a dot above and below, or 
it can be realized by a diagonal slash as in a fraction or 
as a horizontal bar as in a large fraction. Any occurrence 
of one of these will have associated with it the canonical 
form of division so that, in successive expressions, 
division realized in one way will correspond to division 
realized in another. This correspondence can guide the 
parser to the right interpretation. 

There is also the notion of a canonical expression. 
For example, let's say the parser has already encountered 
the expression A + B and now encounters A +. It maps A to 
the canonical A which has already appeared in previous 
expressions. It will also see, among other things, "A + B" . 
This will motivate the parser to look for B on the oti.er 
side of the +. This is one way that expectations are 
generated. The advantages of looking for common 
subexpressions are that an expression which has been written 
clearly and carefully once, can then be written more 
sloppily and the parser will be more robust and able to 
recognize it . 

There are even stronger semantics to guide parsing. 
Another level of equivalence that the parser can look for is 
al<-ebraic equivalence under the legal transformations of 
algebra: cancellation, solviniT equations, dividing through 
both sides of an equation, etc. Through the series of steps 
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in solving an algebra problem, the variables and expressions 
can be traced from line to line, transformed slightly but 
largely remaining the same. There is hope that the parser 
can do this in conjunction with the understander which is 
monitoring a student. The trouble that the parser has may 
be trouble that the student has, and may be worth commenting 
on . 

We've said that the character recognizer used in the 
Lewis system will probably be used in the Purcell system. 
It has, however, a few defects. The recognizer is a feature 
extracter and the features that it extracts are not 
sufficient for distinguishing some characters that it should 
distinguish. The recognizer confuses u's and v's, 2's and 
z's, c's and parentheses, 7's and parentheses , 1 ' s and 
parentheses, integral signs and 1's, etc. One feature that 
probably could be added to the recognizer that could be very 
useful in distinguishing many of these otherwise ambiguous 
characters would be some kind of inflection point 
recognizer, some second derivative of the curve, or some 
measure of curvature to recognize sharp corners of v's, 7's, 
etc. Extra feature extraction may be desirable after an 
initial guess at the letter has been made. If the letter 
falls into a certain kind of ambiguity, a second feature 
detector might be invoked. Alternatively all the features 
might be extracted initially on all the strokes. Either 
method would work. 

Where are we in the construction of the Purcell system? 
Most of the system is still on the drawing board. What has 
been implemented is a program to grow a network of 
two-dimensional neighbor relations between characters. By 
using the enclosing rectangles of pairs of characters which 
are near each other and locations of their x and y centers, 
the pairs are put into spatial relations such as above or 
below. The first relation builder is being interfaced with 
a tablet to see how it can handle simple expressions. The 
relations are easy to find for very simple, carefully drawn 
data and it is expected that it will degrade gracefully, 
still finding most relations correctly. Some of the partial 
strategies have been experimented with by hand. Samples of 
data from a large number of subjects have been used to 
design the ::ystem described above. A lot of implementing 
lies ahead, but there is every reason to expect that the 
parser will be able to understand algebra expressions from 
the tablet. The open question is how much the user will 
have to adapt to the computer to make the task easier for 
it. We hope that as few conventions as possible will be 
necessary so that the tablet can be as natural and 
unobtrusive as possible. We hope, for example, that 0»s or 
z's will not have to be crossed to distinguish them from o's 
and 2's. Perhaps this convention can be used as a backup in 
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the absence of context, but we hope the user will not view 
the tablet as any different than paper, 

'['here are many things that can be done, with the 
dynamic medium of a graphics display and a tablet, that 
can^t be done with pencil ,ind paper. For the time being, we 
plan to avoid these so that the tablet can be as similar to 
paper as possible. It is tempting to put in some kind of 
editing facilities to the tablet, but this will not be 
pursued . 

We have high hopes that the tablet will be a large 
improvement over typewi^iter input of algebra. We think that 
tablet understanding will touch on many interesting issues 
ir: its own right; issues of using context to disambiguate a 
noisy environment, and developing more general notions of 
parsing. Related to this graphical input is the graphical 
output of expressions. The Lewis system also included an 
expression "pretty printer" that transformed expressions 
into two-dimensional text. It handled layout successfully, 
but had trouble expressing superscripts and subscripts 
because characters had to lie on a line or clearly up on the 
next line above and subscripts could not be made smaller. 
Perhaps characters could be scaled and translated to form 
nice expressions. Some expressions or characters could be 
transformed by rule, not merely by shrinking or stretching. 
Square roots for example have a definite width and height 
and could merely have a rule to extend the root sign. We 
don't see it as important to rewrite the student input for 
feedback. If the pr.rser works, there's no need for it. The 
original input is most easily recognizable to the user; 
after all it's what he entered. But there is a time when 
the nutor would like to give examples, and give them in as 
sirr^ilar a way as possibl- to the way the student would 
perform the same example. For instance, a demonstration of 
canceliinp; fractions needs an expression printer. 

In conclusion, we see the graphical medium for both 

input and output as having the potential for greatly 

improving: the communication between computer tutor and 
student . 
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CHAPTER 6 
SYSTEMATIC UNDERSTANDING 

Synthesis, Analysis and Contingent Knowledge 

in 

Special ized Understanding Systems* 



A fundamental challenge in the design of useful 
knowledge-based CAI systems is to find ways to embed 
knowledge and procedural skills into these systems in a 
manner which renders them both efficient and robust 
understanders (e.g. problem- solvers , quest ion- answerers , 
student modeller."^, answer evaluators, etc.). The designers 
of nearly all generative and/or Al-based CAI systems have 
avoided facing the complexities inherent in substantive 
domains of knowledge by tackling only small, well-defined 
and closed subject domains. Indeed there are few, if any, 
knowledge-based systems in existence that honestly face the 
challenge of completely modelling a realistic body of 
knowledge! This is due in part to the shortcomings of 
current computer technology (which is rapidly and 
dramatically improving) and in part to the lack of any 
comprehensive and viable theory of how large and complex 
bodies of knowledge can be represented in computer-based 
s ys tems . 

As an attempt to remedy this deficiency we have 
combined the collective experience gained in building 
various knowledge-based CAI systems, and have constructed 
the beginnings of a theory for how to represent complex 
bodies of knowledge. It is our intent that a fully 
developed theory will not only establish some guiding 
principles for how to build practical CAI systems that 
really can "act" as intelligently as a human tutor, but that 
such a theory will also provide us with a new and more 
powerful methodology for understanding ohe structire and 
content of the particular knowledge needed to carry out a 
collection of tasks. If so, we will have a new handle on 
how to view the skills that must be imparted to a student 
and how the student is to use this knowledge in executing 
his assigned tasks! 



(*)This chapter is to appear, in a slightly altered form, as 
a chapter in Representations and Understa n ding — Studies in 
Cogni t ive Sc ience edited by D. Bob row and A. Collins, 
Academic Press 1975. ^ _ 
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In the design of a knowledge-based system, careful 
attention must be paid to the choice of representations for 
different components of that knowledge. Clearly, the best 
representation for a body of knowledge depends on how that 
knowledge is to be used by the program, and thus better 
characterization of the uses of knowledge is likely to lead 
to better ways of designing knowledge representations. In 
this chapter we present the SCA model , a framework for 
describing the structure of "conceptually efficient" 
understanding programs, based on a characterization of three 
fundamentally different ways in which knowledge is used in 
such programs. Even though the SCA model is not full- 
developed, we feel that it can be of use both to those who 
are designing understanding programs, and to those who wish 
to study existing programs to develop insights into 
different approaches to representing knowledge. 

The version cf the SCA model described in this chapter 
applies to a class of programs that we refer to as 
soeuialists or exper t undsrstan der s . These programs 
understand the world in the sense that they can take in a 
collection of data describing some situation, and then 
answer questions about that situation. The programs are 
referred to as specialists because they are only able to 
deal with a limited class of situations, and can only answer 
questions in some limited domain of specialization. 

We are still in the process of developing the SCA 
model, and many ideas are still in the form of 
speculations and intuitions. In order to present 
the essential aspects of the SCA model in the 
clearest possible form, we have described a 
simplified version of the model which does not 
deal with a number of crucial issues in the design 
of understanding systems. In particular, though 
the systems to be discussed "learn" about their 
environment in the simple sense of taking in 
descriptions of the current state of \.he 
environment, they do not learn how to improve 
their oerformance, nor do they extend their 
initial knowledge about those properties of their 
environment which hold in all states. 
Additionally, though we believe strongly that 
complex problems in understanding will be solved 
by programs built from many specialized modules 
that communicate and interact within a complex 
control structure, the model we describe consists 
of two programs r.hat interact by simply creating 
and accessing a common data-structure. 
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Part cf our intent in articulating the SCA model is to 
present a point of view on the design of export 
understander s which provides insight into how knowledge of 
the expert's domain can be effectively used to design a 
conceptually efficient understander . The SCA model provides 
a framework for studying the structural similarities of a 
variety of superficially dissimilar high-performance 
understanding .systems including perceptual understanding 
systems, natural language data base management systems and 
question answe»^ing systems. It also provides a basis for 
discussing many current knowledge-based research issues such 
as the meaning of analogical representations, the pros ani 
cons of different inference schemes (e.g. computational, 
rule-driven...). ad hoc versus general knowledge 
representations, and the integration of such tools as 
simulation into knowledge-based systems. 

The r ^maindor of this chapter is divided into two major 
sections. The first section is an account and explanation 
of the concepts involved in the SCA model. This section 
starts with a orie.'' description of the general concepts of 
the SCA model, and then gives three examples of systems so 
organized, in order to provide an intuitive feel for the 
meaning of the concepts and their intended breadth. The 
second section describes various ways in which the SCA model 
can give insighc into problems in the design of knowledge 
representations. 
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SecLLon 1 



THE SCA MODEL 



Brie f nescript io n of the SCA Mod el 

For the purpose of providing an overview of this model, 
let us consider a hypothetical expert understander who 
obtains information about the world in the form of a 
collection of basic uninterpreted "sensations" or raw input 
data. The expert must answer questions about the world on 
the basis of this raw data. Much like a human expert, our 
hypothetical expert uses his specialized knowledge to 
combine, organize and augment this data, and thereby 
synthesize a substantially enriched world model or 
contingent knowledge structure (CKS). 

In its simplest form, this CKS might just be a 
data base of assertions describing the expert's 
current knowledge. In general, however, the CKS 
is a complex data structure that represents the 
expert's current model of the world. This model 
may include analog representations of some aspects 
of the world, as well as semantic networks and 
other knowledge representations. 



The SCA model extends this idea and specifies tha.t an 
understander should be designed as two separate experts, one 
to synt hesize a CKS from the collection of raw input data 
and the other to analyze the information in the CKS in order 
to answer the questions posed to the understander. The raw 
input data are the result of the operation of other programs 
or input devices, and are not usually represented in terms 
of concepts that are directly usable by the understander. 
The first expert, or svnthesist . converts this raw input 
data into a form suitable for the understander to act on. 

In addition to simply augmenting the information 
contained in the raw data, and reorganizing this 
data, the synthesis operation often changes the 
elementary concepts in which the information is 
expressed (e.g. changing from a representation of 
a visual scene as an array of intensities to a 
collection of boundary lines, or from a collection 
of boundary lines to a set of three-dimensional 
objects) . 

The seoond expert, or analyst . retrieves knowledge 
explicitly represented in the CKS, and uses world knowledge 
to inf-r facts not explicitly contained in the CKS, perhaps 
using procedures as complex as general theorem provers. 
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The ef fec^llveness of the total understander depends 
critically on the designer's ability 1) to provide the 
correct balance of skills (and computational load) between 
the synthesist and the analyst, 2) to design a class of 
CKS*s which represent Just those aspects of the world state 
needed to facilitate the operation of the analyst, and which 
can be directly represented as efficient combinations of 
procedures and data-structures, and 3) to use special 
purpose techniques to enable the synthesist to fill in the 
CKS in an efficient way. 
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Section II: 

GENERA L DESCRIPTION OE THE ELEMENTS OF THE SCA MODEL 
The Syn t ho >st 

The synthos.i3t deacribes or accounts for a collection 
of input data in terms of some acceptable structure which is 
an instance of a class of structures specified by the 
designer. It must impose the "best" acceptable structure on 
the input data (which may al ready be organized in some 
fashion). The imposed structure accounts for the original 
input data in terms of concepts useful to the analyst, and 
provides the only mechanism through which the analyst is 
permitted to obtain information about the state of the 
world. In general, the analyst never has direct access to 
the information contained in the raw input data. In more 
elaborate versions of the SCA model the analyst can request 
the synthesist to synthesize a new CKS based on the current 
needs of the analyst. This may change the effective 
information that the analyst obtains from the raw input 
data, by means of the CKS, but it does not remove the CKS as 
a necessary intermediary. 

There are three inter-related operations that may be 
performed by the synthesist: 

1) st r uc ture determinatio n - determine which of 
the potential structures (e.g. configurations of 
blocks, parse trees, etc.) it knows about should 
be used to represent the given raw data, 

2) matching - determine the correspondence between 
the raw data items and the parameters of the 
chosen structure, and, 

3) augmentation - determining parameters of the 
CKS not directly corresponding to raw data items, 
but which must be chosen to satisfy some set of 
constraints . 

These operations may require a coordinated search through 
both the entire set of facts and the possibly infinite set 
of potential structures, in a matching or parsing phase. 
The synthesist may put the given facts in a canonical form, 
for example by algebraic simplification, by reduction to^ a 
structure composed of semantic primitives, or by using 
hash-coding and search to reduce an expression to a unique 
internal structure <Reboh 73>. Finally, the synthesist may 
fill in pa^'ar-eters of the structure in a manner determined 
by the i input data and a set of constraints. The 
synthesist may orly have to perform one or two of these 
operations (for example, when the matching operation and 
structure determination are trivial but the augmentation 
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o pe ra t i o n i s d i f f i < : m 1 t ) 



It. i.s often ponyi.t)lo to d ir,t LriKui repositories of 
knowledge correapondinR to an active oynthesis agent, a 
description of the c 1 a s s oV por^sible contingent knowledge 
structures, and a goodness measure which evaluates how well 
a structure matches up with the raw input data. In the 
LUNAR parser <Woods 73>, for example, the description of the 
class of possible contingent knowledge structures (as well 
as rules for searching for the "best" structure) is given as 
an Augmented Tranr.ition Network and the active agent is the 
general ATN par:'.er. In many cases, however, it is 
impossible to separate the synthesizing agent from the 
description of the structures to be synthesized. In these 
cases the synthesist is implemented as a specialized 
procedure for instantiating a particular class of structure 
- this is often mo re efficient, t hough less flexible. 

It may seem possible to build in accessing functions in 
the analyst and do away with synthesis, particularly where 
the information used by the analyst only depends on a small 
amount of raw input data. However, the choice of structure, 
even for a small portion of the raw input data, often 
depends on the entire collection of raw input data. In 
these cases one cannot do without the synthesist by simply 
putting extra processing in the analyst - the extra 
processing needed is actually the complete synthesis 
operation. For example in the blocks world the analyst need 
only look at a structure determined by a small part of the 
input to answer questions about a single block, but the very 
concept of a block depends on a synthesis process that takes 
into account the entire set of line segments in the scene. 



Contingent Knowledge S"* r uc t ur es 

The contingent knowledge structur e , ( CKS ) is a 
data-structure which represents the understander * s knowledge 
of the state of the world. The CKS forms the interface 
between the synthesist and analyst, and thus determines the 
way ' which the characteristics of the world are 
inte ^ted by the specialized question- answering and 
prob. -solving capabilities of the analyst. Because of 
this, the capability and efficiency of an SCA understander 
depends critically on the design of the CKS and its computer 
representation , 

There are two distinct aspects to the design of the 

CKS: 

1) the conceptual structure which the CKS imposes 
on the world 
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2) the dnt.a s t. r U(< t ut'Ois aiKi pt-oood ur om choiu?n t.o 
represeni. t.hc onLitio.s , propetM, iea and relation:; 
which mak(^ up the ooncoptunl ntructure of a CKS. 



Like a languaK'N t.^u':' clas::; of possible CKS ' s dofirir a 
set of conceptual ontitios, relations and structures which 
determine the way the analyst can must easily express its 
questions and operate on its model of the world. It 
determines the distinctions which the analyst can possibly 
make between states f the world. For example, in the 
blocks world understan the conceptual entities are blocks 
with sizes, shapes, oositions in three-dimensional space, 
and relations depending on their positions - rather than 
corners, connected sequences of line segments, or any other 
possible structural entities. 

Given several alternative conceptual structures for the 
class of CKS's, it is tempting to choose the "most 
expressive" one, to facilitate the description of the states 
of the world and the questions of the analyst. However, the 
analyst must be capable of dealing with any CKS within the 
chosen conceptual framework, and more subtle and complex 
frameworks require more complicated (therefore usually less 
efficient) analysts. Thus there is a tradeoff between 
expressiveness and efficiency, and a good conceptual 
structure is one that can readily express those properties 
of the world that are relevant to the analyst, but does not 
lend itself to describing irrelevant details or impossible 
states. An example of an unproductively rich conceptual 
structure for the blocks world CKS would be one that could 
represent all connected sets of line segments, including 
r-ollections of lines not corresponding to the boundaries of 
a set of blocks. Such collections are impossible in the 
blocks world and hence are irrelevant to the blocks world 
analyst . 

We believe this tradeoff is closely related to one 
hypothesized by Dr. Frederick Thompson of Caltech <Thompson 
72, Randall 70, Greenfeld 72 > . Thompson's conjectured 
"Fundamental Theorem of Information" says, roughly, that 
given a collection of observations of the world, and a 
sequence of progressively richer languages, there is an 
intermediate language in which the descriptions of the 
observations carry the greatest information. This most 
informative language is not rich enough to express all the 
details of every observation - the concepts that make up its 
semantics are broader and more abstract than the details of 
the observations, and thus it captures the i^^.portant 
properties of the observations without _ allowing the 
expression of unnecessary or irrelevant detail. 
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!<v I'l it . \ \\r [>ri)V l.tit* onl y rriJMih.'in i :;ffi f'oi' l()t» 

itia 1. y..> t,o r-r!'rT' \a) \ \]c (»ropor t i or> of t.h<^ wcu'ld ".i,.'it,»'. Tin- 
C.K:\ roMiuM*:; t,h(^ i:*)mpl t^x [ t,y of the nnalyst. by [)ro v i <1 i n^'; :\ 
r > f^m o {' onruM) i 1 r ^M)f^ (::'on l:\ t. i o n \\) r won 1 d s la i .^^ :; , l]t; \ t.o s 
of* i. ht^ world which oan l.rraUMi nimi. larly l~)y tho analyr. 
art* m<']p[KMl into :urnilar CK.S dat 'i -•triK^tur'o.'^ . Thus, t i.^^ 
o-...'tior pt iia I MlrMioture mu.*;l: ho t^iM^^l t;o f*ao i. 1 L t. at.c^ botJi ! iie 

ox[)rension of* t.ho quo.stJ.on.s whioh the /nialyii^t mu:;L answ-u-* 
and the oporalions which t lio aiialv:;.t rniist perform in r:^r :r 
I o a t KS w V V the q u 0 5. t i o n : , 

!- liev" 'hit, in morA. oa:':e3 th^ CKS is b(\^>t viewed as 
'•i model (^or Ihe state oV the .urld, rather than a 
description of tli^- state, so that t le operations of the 
■uialyst correspond more to observations of the world than to 
manipulation of the representation of assertions. We use 
the term ^'observation" in the :w^nse of an operation on the 
w'M'ld that produces information as a result, and whlcu does 
not .'hani^^e tht^ state of the world (this is in accord with 
the simple version of the SCA model, but is not a general 
restriction). We do not mean to imply that such 
observations are simple operations, or that the analyst is 
s, imply an information retrieval meohanism that observes what 
is explicitly present in the CKS. 

Much of^ the und er sta n der * s knowledge of the state of 
the world is not contained explicitly in the CKS, but is 
embedded in a set of tacit agreements between the synthesist 
and analyst as to the way in which the data-structures that 
form the ZKS are to be interpreted. For example: 

a) A linear sequence of links between sev^-ral 
items can be interpreted as a transitive relation 
if tho analyst determines that two items are 
related by seeing if there is a chain of links 
joining them. 

b) A s-:t of po inters to objects from elements of a 
three-dimensional array is sufficient to represent 
many the three-dimensional relUions among a 
group of physical objects if both the synthesist 
and analyst interpret the existence of a pointer 
from an array element A(I,J,PO Lo an object 0 as 
meaning that 0 is lies within a box Postered at 
coordinates ( I , J , K ) . 

c) A list structure can be used to represent a 
procedure for answe r i ng a question, given a set of 
rules such as those used to interpret LISP forms. 
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Given a particular choice of conceptual structure and a 
set of operations (defined in terms of the conceptual 
structure) which the analyst must perform, the designer must 
choose a collection of computer data-structures to represent 
the CKS, which maintains the comprehensibil ity of the final 
program and its overall efficiency. The distinctions we 
draw between the conceptual structure of the CKS, the 
implementation of the conceptual constructs in terms of 
programming language constructs, and the eventual 
implementation of these constructs in terms of machine- level 
primitive operations are an attempt to deal with the problem 
that Hayes <Hayes 74>. poses: 

"a representation which appears to be a direct 
model at one level ... may ... be itself 
represented in a descriptive fashion, so that it 
• becomes impossible to describe the overall 
representation as purely either one or the other. 
It seems essential, therefore, to use a notion of 
level of representation in attempting to make this 
distinction precise." 



The Analyst 

An analyst derives information from a CKS in order to 
answer questions posed by some other process. Informally, 
if the CKS represents a state of the world viewed from a 
perspective defined by the conceptual structure of the CKS, 
then the analyst infers answers to specific questions about 
the state of the world using information in the CKS and a 
set of rules. These rules include laws of the world and 
laws of logical inference, so that the answers provided by 
the analyst correspond to true propositions about the state 
of the world. The CKS is of necessity a finite collection 
of information, but the set of questions one can ask about 
the context usually come from an infinite set defined by 
linguistic rules. Thus, the an alyst is an inference 
mechanism for bridging the ga^ between the finite set of tM 
properti es of the world which are explicit ly represented in 
the CKS^ and the infinite :.et of valid assertion s (answers 
to questions) about the world. which are implicitly 
determined by the CKS . 

The simplest analytic operations consist of the 
application of compositions of functions and relations to 
elements of a CKS, for example forming the sum of the 
lengths of several lines in a geometric structure, or 
comparing such a sum with the distance between two points. 
More complicated analytic operations might consist of using 
the results of such simple operations, along with general 
world knowledge to deduce further properties of the world 
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state represented by the CKS. in many leases the 
understande r need not explain or expl ic itl y just i fy j^ts 
answer s y so explici t logical deduction can be repla ced b^ 
other form s of in ferenc e or computa t ion . For exHrn[)le, in a 
CKS which consists in p.-^.^t of a semantic network, properties 
of an element can be inferred by tracing a path to some 
other element and then applying simple computational rules 
to a description of the relations in the patli and the 
properties of objects on the path. 

The fact that "John's uncle is Jane's grandfather" 
could be derived from a chain "John SON -OF Peter 
HUSBAND-OF Mary SISTER-OF Isaac FATHER-OF Ellen WIFE-OF 
Jack FATHER-OF JANE", by the application of a set of 
simple rewriting rules to the sequence "SON-OP 
HUSBAND-OF SISTER-OF FATHER-OF WIFE-OF FATHER-OF". 
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Section III: 

COMMENTS ON REPR ESENTATION ISSUES 
Universal and Ad Hoc Contingent K nowledge Representation s 

A theory such as the one which we hope underlies our 
SCA model would make it possible to discuss rationally the 
representation of knowledge at the interface of the 
synthesist and analyst, as well as the design and operation 
of synthesists and analysts for particular problem domains. 
It would permit us to phrase meaningful questions about the 
relative merits of different contingent knowledge 
representations from the point of view or efficiency of 
synthesis, analysis and original design, and could thus 
clarify the debate over the relative merits of "general 
purpose" and "ad hoc" representations of knowledge. 

Let us take a superficial look at this debate in terms 
of the SCA model. Given that all understanding systems must 
convert their raw input data into data structures used to 
meet goals of the understander , the more goals that can be 
met effectively with a single structure, the fewer times 
must a synthesist be invoked to create another one. 
However, building a single structure to serve many 
independent goal-oriented procedures may be more difficult 
than building several different specialized structures. In 
addition, the improvement in efficiency of the goal-oriented 
analytic operations may be brought about by the availability 
of specialized contingent structures to make up for the 
extra time spent in building them. 

One must also take into account the resources necessary 
to maintain consistency and compatibility among multiple 
representations, or allow for the problems of potentially 
inconsistent actions by different analysts*. These problems 
are particularly difficult to resolve in understanding 
systems which generate and deal with multiple contexts, such 
as planning and problem solving systems. 

One must also evaluate the impact of "generalized" and 
"ad hoc" representations on the problem of designing 
systems. Clearly, having a single, highly efficient and 
effective knowledge representation would substantial ly 
reduce the time necessary to design new understanding 
systems. Even a unified conceptual framework like "the 
omega order predicate calculus" (as in QLISP) can ease the 
designer's task. On the other hand, the usefulness of 
engineering handbooks attests to the fact that an organized 
collection" of specialized structures with capabilities and 
limitations clearly spelled out can be quite as good a 



(*)There is certainly evidence that human understanders have 
inconsistent representations of knowledge, and that they can 
come to inconsistent conclusions by using different 
techniques for solving problems or answering questions. 
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design aid as a single generalized technique. 



Contingent Knowledge Structures and the 

Antecedent/ Consequent Boundary 

In building understanding systems with procedural 
representations of knowledge, there is a serious design 
problem in the distribution of expertise between the 
antecedent ("if-added") and consequent ( "if-needed" ) 
procedures (as in PLANNER, <Hewitt 72>, CONNIVER, <Sussman 
72> or OLISP <Reboh 73>. Roughly speaking, if-needed 
procedures are triggered by the introduction of specific 
goals and sub-goals to be met by the understander , while 
if-added procedures are triggered by addition of new facts 
to the contextual knowledge base. 

The if- added procedures clearly correspond to the 
operations of synthesis. It may not be clear that 
if -needed procedures correspond to a combination 
of synthesis and analysis, and not simply to 
analysis. Simply speaking, the if-added 
procedures correspond to data synthesis, while the 
if-needed procedures correspond to goal synthesis, 
since they replace a set of goals with a structure 
of goals and sub-goals that combine to satisfy the 
original goals. 



One could conceivably avoid the use of if-added 
procedures entirely, by making all procedures goal-oriented, 
and "reasoning backward", so that nothing is done^ until it 
must be done to meet a specific goal. This runs into 
difficulty since it allows for little coordination between 
the goals and the context, so that it is possible to 
generate vast numbers of irrelevant, impossible and costly 
sub-goals. Additionally,- unless the results of sub-goals 
are added to the knowledge base one can needlessly repeat 
many sub-goal computations. 

Alternatively, one can "reason forward", taking the 
contents of the context and applying if-added procedures to 
derive all the goals which could be met given the context. 
I this approach is implemented in an unrestrained fashion 
one can end up swamping the data base in irrelevant results 
before getting around to meeting the specific goals posed to 
the system . 

The concept of a CKS for a class of goals provides a 
handle on the problem of how far to let the if-added 
procedures run. In essence, the if-added procedures become 
"potential goal" directed, as compared to the "specific 
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goal" directed if-needed procedures. The CKS to be produced 
by the if-added procedures stands in for the entire set of 
potential goals. While it may be said that if-added 
procedures are always directed towards satisfying a set of 
potential goals, the explicit design of a CKS for a given 
set of goals provides a mechanism for keeping track of 
decisions as to what knowledge is to be encoded in the 
if-added as opposed to the if-needed procedures. Given a 
set of goals, one needs only to define a class of CKS to 
or.:anize contextual information and simplify the execution 
of the correspond ing if-needed procedures. If-added 
procedures then implement a synthesis procedure to build 
such structures from any of the expected contexts. One can 
even package such compatible sets of if-needed and if-added 
procedures into "demon teams" (as in QLISP, <Reboh 73>) 

There is a catch to the above suggestion - how does one 
find a compatible set of if-needed demons whose operations 
are facilitated by a single contingent knowledge structure? 
While we have no general answer to the problem, the 
technique used in system un derstan der s is suggestive. 
Essentially, one replaces the search for a CKS and 
compatible set of if-needed demons with a search for a query 
language in which all the demons* information needs can be 
expressed. Starting with a rough idea of these basic 
semantic entities relevant to the demons (e.g. some 
"objects", "structures", "relations", etc.) one. considers 
the types of questions about such semantic entities whose 
answers would help meet the demons* goals. This can often 
be refined to a well-defined set of primitive questions and 
composition operations which can be used to answer all of 
the needed questions. One can then design data structures 
and procedures that facilitate answering all the questions 
in the query language, and the data structure and procedures 
form the class of CKS*s. 



KiRh er-Level Struc tures ( Frames) and World Knowledge 

The raw input presented to an understander is 
insufficient to tell the understander all it needs to know 
about the specific situation it is in. There is always a 
need for world knowledge in the understander to fill in the 
meaning of the input. Several chapters in this book discuss 
the problems of how to organize such higher-level knowledge, 
how to find knowledge relevant to a given collection of 
inputs, and how to use this knowledge to provide an 
interpretation of the input. We refer below to the process 
of combining higher-level knowledge with input information 
as frame-instantiation, and call the resulting structure an 
instantiated frame (even though the higher-level knowledge 
may be a script or scheme, etc.). 
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The SCA model presonted in this paper .does not say much 
about finding knowledge that is relevant to a given 
situation, but it does say something about 
frame-instantiation and the use and structure of 
instantiated frames. In fact, the process of 
frame-instantiation is a synthesis operation, and the 
resulting instantiated frame is a CKS. The primary use of 
the instantiated frame is to provide the information needed 
by analysis procedures, in the best organized form. 

The view of parsing and perceptual processing as 
synthesists (and hence cousins to frame instantiation) leads 
to the realization that different instantiations of the same 
frame can be as different in structure as two distinct 
sentences, or two distinct arrangements of blocks. Many 
descriptions of frame instantiation give the impression that 
'ill instantiations of a given frame have similar structures, 
differing primarily in values that fill in slots in the 
frame. Since slots can be filled by instantiated frames 
this can indeed lead to a structure as complicated as a 
parse tree, but not obviously to a network-structured entity 
like a model for a collection of blocks. Simply expanding 
slots into subordinate frames i: equivalent to top-down 
parsing. For complicated structures, a combination of 
top-down and bottom-up approaches may be advisable, and one 
might usefully apply many of the techniques of natural 
language parsing, including well-formed substring tables or 
charts for handling local ambiguities until they are 
resolved by more global constraints. 

The use of synthesists for relaxation techniques and 
other methods for simultaneous constraint satisfaction 
extends the range of frame-instantiation operations beyond 
those commonly considered. Much attention has been given to 
the problem of determining which frame is to be 
instantiated, and how to switch from attempting to 
instantiate one frame to the instantiation of another, frame 
when difficulties arise (see the working paper by 3. 
Fahlman quoted in Minsky 1975). This suggests that choosing 
the correct frame is difficult, but instantiating it is 
simple. Once one realizes that frame instantiation may 
involve complicated simultaneous constraint satisfaction or 
expansion of structures as complex as natural language parse 
trees, it is clear that complicated synthesis techniques are 
needed for frame instantiation. 

The literature on frame instantiation, and the 
structure of frames, gives little idea about the uses to 
which instantiated frames are put. The SCA model suggests 
that it is vital to know the questions which are to be 
answered with the help of the instantiated frame. A 
synthesist, CKS and analyst are designed together, as 
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interdependent modules, suggesting that both frame 
instantiation and the structure of instantiated frames must 
take into accoun t the set of o p eration s to be performed on 
the instantiated frame . The conceptual structure and 
machine representation of a CKS depend heavily on the 
expected inputs to the synthesist, on the design of the 
synthesist, and especially on the operations to be performed 
by t he analyst . Even when the input s and their real-world 
meaning are fixed (e.g. where the inputs are always line 
segments corresponding to the edges of blocks) the 
information in the CKS must depend on the questions to be 
asked by the analyst (e.g. do they ever refer to position, 
volume , e tc . ) . 
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Section IV: 



CONCLUSION 



The straightforward SCA model presented above is not a 
complete description of our current concept of the design of 
an understanding system. A more complete one is hinted at 
in the section on the LUNAR question-answering system, in 
which the synthesist is itself viewed as an expert which is 
broken down according to the SCA model. In general we hold 
a belief similar to that of Hewitt, in which programs are 
composed of interacting active procedural elements or ACTORS 
<Hewitt 73>. We feel that individual ACTORS should each be 
organized in terms of the SCA model, with separate 
synthesis, analysis and contingent knowledge structures. 
The synthesist (or analyst or CKS) of a more complicated 
ACTOR is built up by the activity of a collection of 
cooperating ACTORS. The crucial issue then becomes the 
design of the sociology of ACTORS, that is, the 
communication and control strategies used to organize the 
efforts of the independent ACTORS. The SCA model itself is 
a partial (very partial) answer to this organization 
problem, since it suggests that the ACTORS composing a 
complex ACTOR are organized into three separate groups that 
interact by well-defined means. We believe that another 
valuable source of ideas for the sociology is the work of R. 
Kaplan <Kaplan 75> on the GSP natural language system, in 
v/hich the components that make up a parser are organized 
into consumer and producer modules that interact with one 
another . 

Given the changing economics of machine architecture, 
in which it is becoming possible to think of machines with 
hundreds of interconnected micro-processors, the ability to 
view AI processes in a way which leads to parallel 
decomposi t iOiis may be quite useful. Viewing synthesis as a 
constraint satisfaction operation leads naturally to 
implementing it by groups of parallel processes which 
cooperate to find the best structure to match the given set 
of constraints. We should point out that the economics that 
we are approaching is not a new one - it is the economics of 
genetic systems, the economics of constructing a brain. 
Given the information needed to define one type of neuron 
and its pattern of interconnections, it is not substantially 
more difficult to grow millions or^ billions of copies. Many 
of the underlying intuitions that led to the SCA model stem 
from a study of the interactions of neurons in terms of a 
model for neuron function suggested by Dr. J.Y. Lettvin 
<Lettvin, personal commun ioat ion> . We hope- that it will 
someday be possible to unify these disparate sets of ideas. 
Possibly some of the ideas arising from extensions of the 
SCA model to arrays of interacting SCA modules may be useful 
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as way of viewing the operation of the brain. 
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